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IV 


ABSTRACT 


This  study’s  purpose  was  to  identify  opportunities  to  improve  the  P-8A  test 
working  description  (TWD)  process  in  support  of  the  P-8A  test  program.  This  study 
researched  other  NAVAIR  test  programs  that  used  a  contractor  and  government 
integrated  test  approach.  This  study  determined  the  proper  balance  of  improvements  to 
support  the  approval  of  TWDs  to  keep  pace  with  the  testing.  The  process  was 
functionally  decomposed  to  look  for  process  redundancies,  choke  points  and  out-of¬ 
sequence  sub-processes.  Two  changes  were  identified  and  implemented  that  reduced  the 
process  from  five  phases  to  four  phases. 

Corel’s  iGrafx  software  was  selected  to  model  the  process.  The  model  varied 
TWD  development  time  and  test  team  resource  level  to  analyze  alternative  TWD 
development  process  concepts.  Alternatives  examined  excursions  from  a  baseline 
condition  that  was  limited  by  the  existing  team  resource  level  (the  lower  bound)  to  an 
upper  bound  that  assumed  unconstrained  resources.  The  upper  bound  conditions 
represented  the  earliest  time  the  TWDs  could  be  completed.  Additional  alternatives  were 
analyzed  until  an  “optimal”  resource  level  was  found  that  would  support  the  test  program. 

It  is  recommended  this  type  of  modeling  be  applied  to  other  test  programs  and 
systems  engineering  processes  to  improve  their  efficiency. 
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EXECUTIVE  SUMMARY 


The  purpose  of  this  study  was  to  identify  opportunities  to  model  and  improve  the 
P-8A  test  working  description  (TWD)  process  that  would  support  the  P-8A  test  program. 
This  study  researched  other  NAVAIR  test  programs  that  used  a  contractor  and 
government  integrated  test  approach.  These  test  programs  were  the  V-22,  F/A-18E/F, 
and  the  E-2D.  The  three  programs  experienced  similar  challenges  as  they  progressed 
through  test  execution.  The  P-8A  program  has  already  incorporated  many  of  their 
lessons  learned  associated  with  defining  roles  and  responsibilities.  The  research  of  these 
programs  also  found  each  program  struggled  with  test  plan  throughput.  Their  solutions  to 
this  problem  varied  across  programs  and  made  it  difficult  to  apply  lessons  learned  in  this 
area  to  the  P-8A  program. 

The  challenge  for  this  study  was  to  detennine  the  proper  balance  of  improvements 
to  support  the  production  and  approval  of  TWDs  to  keep  pace  with  the  testing.  This 
required  further  analysis  of  the  P-8A  TWD  process.  A  software  model  of  the  process  was 
built  to  analyze  various  scenarios  to  identify  the  balanced  solution.  To  look  for  process 
redundancies,  choke  points,  and  out-of-sequence  sub-processes,  the  study  applied  value 
stream  mapping.  After  completing  the  initial  functional  decomposition,  two  additional 
changes  were  identified  and  implemented,  which  reduced  the  process  from  five  phases  to 
four  phases.  Following  the  functional  decomposition,  a  computer  model  of  the  process 
was  developed.  Process  changes  based  on  the  modeling  have  improved  the  quality  of 
TWD  written  reports.  As  the  additional  staffing  is  applied,  in  addition  to  other  changes 
suggested  by  the  modeling,  process  time  should  also  improve  significantly. 

This  study  used  the  software  suite  Corel’s  iGrafx  Process  for  Six  Sigma  to  model 
the  process.  By  varying  TWD  development  time.  Integrated  Test  Team  resource  level, 
and  by  re-sequencing  sub-processes,  alternative  TWD  development  process  concepts 
were  analyzed  using  the  process  model.  Examined  alternatives  ranged  from  a  baseline 
condition  that  was  limited  by  the  existing  January  2009  Integrated  Test  Team  resource 
level  (the  lower  bound)  to  an  upper  bound  that  assumed  unconstrained  resources.  The 


upper  bound  conditions  were  chosen  to  determine  the  earliest  time  the  TWDs  could  be 
completed.  The  study  also  analyzed  other  alternatives  whose  conditions  were  in  between 
those  boundary  conditions.  Through  this  analysis,  it  was  possible  to  identify  an 
“optimal”  resource  level  to  support  the  test  program,  identified  as  Alternative  B. 
Alternative  B  set  the  model’s  input  parameters,  so  the  TWDs  were  delivered  at  a  rate  that 
was  just  ahead  of  the  flight  test  schedule  need  dates,  thus  providing  an  optimal  solution. 
The  selected  alternative  required  the  Integrated  Test  Team  resource  levels  to  increase  by 
approximately  150  percent  over  the  baseline,  thus  identifying  a  severe  understaffing 
problem. 

Application  of  this  model  to  other  NAVAIR  test  programs  and  systems 
engineering  processes  is  underway  and  shows  promise  of  improving  their  efficiency  as 
well. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  P-8A  (Poseidon)  aircraft  will  be  the  next  generation  USN  Maritime  Patrol 
and  Reconnaissance  (MPR)  system,  specifically  intended  to  replace  the  aging  P-3C 
systems.  The  P-8A  will  have  systems  that  provide  capabilities  to  perform  the  current  and 
future  missions  for  the  Maritime  Patrol  and  Reconnaissance  Force  (MPRF). 

An  Analysis  of  Alternatives  (AoA)  for  the  MPRF  was  completed,  and  a  final 
report  was  accepted  29  May  2002.  The  report  identified  manned  aircraft  as  an  essential 
element  of  the  suite  of  systems  that  will  satisfy  the  operational  requirement  (CJCS, 
2003).  The  P-8  A  program  underwent  a  19-month  Component  Advance  Development 
that  resulted  in  the  selection  of  the  Boeing  militarized  737  in  September  2002.  The 
Milestone  B  Defense  Acquisition  Board  (DAB)  review  conducted  28  May  2004  approved 
the  P-8  A  program’s  entry  into  the  System  Development  and  Demonstration  (SDD)  phase. 

Since  entering  into  the  SDD  phase,  the  P-8A  program  has  completed  several 
program  milestones.  It  completed  Preliminary  Design  Review  in  November  2006  and 
Critical  Design  Review  (CDR)  in  June  2007.  The  two  most  recent  program  milestones 
were  the  Test  Readiness  Review  (TRR)  in  May  2009,  closely  followed  by  the  First  Flight 
Readiness  Review  in  August  2009.  The  first  P-8A  scheduled  for  flight  test  arrived  at 
Naval  Air  Warfare  Center,  Aircraft  Division  (NAWCAD),  Patuxent  River,  in  late  August 
2009.  Within  the  P-8A  flight-test  program,  there  are  approximately  72  test  plans, 
otherwise  known  at  NAWCAD  as  test  working  descriptions  (TWDs),  for  execution  by 
the  NAWCAD  Patuxent  River  test  team.  These  72  TWDs  need  development,  review, 
and  approval.  These  test  plans  describe,  in  detail,  how  to  exercise  the  system  at 
appropriate  points  in  the  operating  envelope  of  the  system  in  an  attempt  to  demonstrate 
the  design’s  capability  to  meet  threshold  and  objective  technical  parameters.  Developing, 
reviewing,  and  approving  these  plans  have  not  always  followed  a  true  standardized  set  of 
procedures. 
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B. 


PURPOSE 


The  TWD  process  is  the  Navy’s  process  to  develop  detailed  test  plans  for  a  flight 
test  program.  This  severely  challenged  process  is  currently  not  well  defined,  and  it  does 
not  provide  the  throughput  necessary  to  meet  the  current  P-8A  test  schedule.  The 
purpose  of  this  study  was  to  model  the  current  P-8A  TWD  process  and  identify 
opportunities  to  improve  and  increase  the  efficiency  of  the  process.  In  addition  to 
modeling  the  TWD  process,  this  study  documents  all  sub-processes,  records  interaction 
between  the  Boeing  and  USN  test  team,  and  identifies  any  shortages  of  critical  skills. 

C.  RESEARCH  QUESTIONS 

This  research  addresses  P-8A  TWD  Process  requirements.  The  subsequent 
chapters  address  the  following  questions: 

1.  Can  modeling  the  P-8A  TWD  process  allow  opportunities  to  improve  and 
increase  the  efficiency  of  the  process? 

2.  Are  there  key  resource  elements  and/or  process  steps  that  influence  the 
efficiency  of  the  process? 

3.  Can  this  simulation  be  applied  to  other  systems  engineering  processes  for  the 
P-8  A? 

4.  Is  there  applicability  of  this  model  to  other  DoD  test  programs? 

D.  BENEFIT  OF  THE  STUDY 

There  are  several  benefits  resulting  from  this  study,  five  of  which  are  listed 
below.  The  first  benefit  is  the  baselining  and  documenting  of  the  NAVAIR  TWD 
process.  The  second  benefit  is  improving  the  efficiency  of  the  TWD  process  for  the  P-8A 
program  so  that  it  may  meet  the  program  milestones.  The  third  benefit  is  a  model  that 
provides  the  proper  sizing  of  the  P-8A  test-team  staffing  and  necessary  resources  to 
achieve  the  required  throughput  of  TWDs  to  meet  the  flight  test  schedule.  The  fourth 
benefit  is  the  ability  to  apply  the  TWD  modeling  tool  to  other  NAVAIR  test  programs. 
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The  last  benefit  is  identifying  other  P-8A  systems  engineering  processes  that  this  tool  can 
simulate  with  minor  model  adjustments.  Processes  that  could  be  possible  candidates  are 
the  risk  management  process  and  engineering  change  process.  These  processes  have 
similar  steps  compared  to  the  TWD  process.  The  TWD  model  can  be  adjusted  with 
minor  changes  to  account  for  their  specific  parameters.  Results  from  the  modeling  can 
begin  to  identify  opportunities  to  improve  process  efficiency  and  reduce  the  existing 
backlog  of  products. 

E.  SCOPE  AND  METHODOLOGY 

This  thesis  focuses  on  baselining  the  current  P-8A  TWD  process,  modeling  the 
process,  and  running  simulations  while  varying  key  parameters  to  identify  opportunities 
to  improve  the  processes  throughput.  Key  parameters  within  the  model  are  test  team 
skill-sets,  availability  of  those  skill-sets,  available  funding  resources,  complexity  of  the 
TWD,  duration  of  each  phase  of  the  TWD  process,  percentage  of  rejected  TWDs 
requiring  rework,  and  availability  of  approval  authorities. 

To  provide  the  baseline  of  the  process,  the  study  used  infonnation  provided  by 
test  engineers,  the  Government  Flight-test  Director  (GFTD),  the  P-8A  Assistant  Program 
Manager  for  Test  &  Evaluation,  Boeing  test  engineers,  the  Boeing  Chief  Engineer,  and 
the  Boeing  Flight-test  Director.  Research  was  conducted  on  other  integrated  test 
programs  whose  approach  was  to  integrate  developmental  testing  and  operational  testing. 
The  objective  was  to  understand  the  challenges  of  integrating  a  test  team  with  both 
Government  and  Contractor  personnel.  The  baseline  process  was  modeled  and 
simulation  runs  were  conducted  to  understand  the  influence  of  each  of  the  parameters. 
The  author  documented  and  analyzed  the  results  to  look  for  improvement  within  the 
process.  Adjusting  the  process  input  parameters,  more  simulation  runs  were  conducted  to 
understand  the  impacts.  By  comparing  the  results  back  to  the  schedule  constraints,  the 
study  aimed  to  determine  the  success  of  the  improvement  choices.  The  author  created 
recommendations  and  submitted  them  to  PMA290  for  incorporation  into  the  P-8A  test 
program.  In  addition,  the  study  conducted  a  review  of  existing  systems  engineering 
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processes  used  within  PMA290  that  could  benefit  from  this  model.  It  provided 
recommendations  to  PMA290  leadership  for  possible  incorporation  to  program 
operations.  Finally,  the  author  forwarded  lessons  learned  from  the  model  development 
and  the  implementation  to  AIR  5.0,  NAVAIR’s  test  community  for  possible  application 
to  other  test  program. 

F.  CHAPTER  SUMMARY 

This  study  begins  with  a  review  of  the  existing  P-8A  Integrated  Test  Team 
structure  and  goals.  The  review  also  includes  researching  the  existing  P-8A  TWD 
process  to  understand  roles,  responsibilities,  work  environment,  and  constraints  between 
the  contractor  and  government.  Chapter  II  provides  an  overview  of  the  P-8A  program.  It 
describes  the  integrated  test  team  approach  and  discusses  earlier  efforts  to  improve  the 
TWD  process.  Chapter  III  describes  how  the  DoD  instructions  define  an  integrated  test 
program  and  the  approach  used  by  other  USN  integrated  test  programs  to  address  their 
test  planning  hurdles.  Chapter  IV  describes  the  selection  of  modeling  software, 
construction  of  the  model,  including  assumptions  and  inputs  to  develop  the  TWD  model 
and  its  validation.  Chapter  V  presents  results  from  the  simulations.  Chapter  V  also 
presents  recommendations  regarding  improved  TWD  process,  the  appropriate  resource 
requirements,  and  applicability  of  this  method  to  other  NAVAIR  test  programs.  Chapter 
VI  provides  the  conclusions,  and  identifies  possible  systems  engineering  processes  in 
PMA290  that  could  benefit  from  applying  the  modeling  technique  and  other  applications 
within  the  NAVAIR  test  community. 
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II.  P-8A  (POSEIDON)  PROGRAM  OVERVIEW  AND  TEST 
WORKING  DESCRIPTION  (TWD)  PROCESS 


A.  INTRODUCTION 

This  chapter  introduces  the  P-8A  program  organization,  the  Integrated  Test  Team 
(ITT),  and  the  test  working  description  (TWD)  process.  A  TWD  in  simple  terms  is  a 
detailed  test  plan  that  describes  the  test  procedures  and  test  points  to  achieve  a  desired 
test  condition.  Test  data  collected  under  a  set  of  conditions  that  is  used  to  verify 
performance  parameters  is  a  test  point.  The  TWD  process  requires  a  significant  amount 
of  time  and  effort  to  position  the  P-8A  program  to  execute  the  required  test  program 
successfully.  A  successful  TWD  process  is  one  that  delivers  approved  TWDs  early 
enough  to  avoid  delays  in  the  actual  flight  test  program. 

B.  PROGRAM  OVERVIEW 

The  P-8  A  will  replace  the  manned  P-3C  maritime  patrol  aircraft  that  currently 
provides  broad  area  Anti-submarine  Warfare  (ASW)  and  Anti-surface  Warfare  (ASUW) 
capability  along  with  substantial  armed  maritime  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR).  The  P-8A  Poseidon  is  principally  an  ASW  platform  that  also 
performs  ISR  missions  as  a  portion  of  an  ISR  Family  of  Systems,  including  the  EP- 
3/EP-X  signals  intelligence  platforms  and  the  Broad  Area  Maritime  Surveillance 
(BAMS)  Unmanned  Aircraft  System  (UAS).  Figure  1  provides  an  illustration  of  a  P-8A, 
which  is  a  modified  737-800  aircraft,  side  by  side  with  the  P-3C. 


Figure  1.  Pictorial  depiction  of  P-8A  and  P-3C 
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The  P-8A  is  a  land-based  aircraft  that  will  conduct  broad-area  maritime  and 
littoral  patrol  and  ISR  to  provide  Maritime  Domain  Awareness  (MDA).  It  will  contribute 
to  the  Common  Operational  Picture  (COP)  during  peacetime  and  during  buildup  of 
tensions  prior  to  commencing  hostile  operations.  The  P-8A  will  provide  responsive, 
worldwide  forward  presence,  engage  allies  and  joint  forces  in  exercises,  contribute  to 
maritime  homeland  security,  and  contribute  to  lesser  contingency  operations  (e.g., 
disaster  relief  or  non-combatant  evacuation).  During  Major  Combat  Operations  (MCO), 
P-8A  will  provide  assured  access  for  Carrier  Strike  Groups  (CSG)  and  Expeditionary 
Strike  Groups  (ESG)  to  establish  and  maintain  sea  bases  secure  from  hostile  surface  and 
submarine  threats.  It  will  also  patrol  and  protect  sea  lines-of-communication  (SLOC) 
against  the  same  threats  while  contributing  to  the  COP  (CJCS,  2003). 

The  structure  of  the  P-8  A  program  organization  has  five  major  teams 
complementing  program  management:  engineering,  integration,  testing,  logistics,  and 
training.  The  five  teams  are  shown  in  the  gray  box  located  in  the  center  of  Figure  2, 
which  shows  the  entire  P-8A  team  organization.  They  make  up  the  P-8A  leadership 
teams  along  with  Program  Management.  The  five  teams  are  broken  down  into  sub-teams 
that  address  the  major  sub-systems,  such  as  the  air  vehicle,  mission  systems,  software, 
product  support,  manufacturing,  testing,  verification,  training,  and  fleet  introductions. 
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Figure  2.  P-8  A  Integrated  Product  Team  Organizational  Chart 


C.  INTEGRATED  TEST  TEAM  CONCEPT 

Historically,  there  has  been  a  difference  between  contractor  and  government 
testing.  Contractor  developmental  testing  focused  on  developing  the  aircraft’s  flight 
envelope.  Government  developmental  testing  concentrated  on  assessment  of 
specification  compliance  as  well  as  operational  suitability.  The  government  testing 
would  assess  the  system’s  ability  to  meet  key  performance  parameters  (KPP)  defined  by 
the  ORD.  The  KPPs  have  a  threshold  and  objective  values.  A  threshold  is  the  minimum 
level  of  performance,  and  an  objective  is  a  desired  level  of  performance.  An  example  for 
a  range  KPP  would  be  800  nautical  miles  as  a  threshold  and  1200  nautical  miles  as  the 
objective.  The  distinction  between  a  contractor  DT  and  government  DT  is  the  contractor 
would  strive  to  test  to  the  threshold  requirements.  The  government  DT  program  would 

strive  to  find  the  actual  capability  of  the  product.  As  the  Integrated  Product  Team  (IPT) 
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concept  matured,  NAVAIR  recognized  that  there  could  be  efficiencies  captured  within 
the  test  arena  by  combining  contractor  and  government  developmental  testing. 

Early  in  the  SDD  phase,  the  P-8  A  program  decided  to  pursue  an  Integrated  Test 
Team  (ITT)  approach  to  execute  the  verification  phase  of  the  program.  This  decision  was 
based  on  recent  successes  of  other  large  aviation  acquisition  programs  such  as  Advanced 
Hawkeye  (E-2D),  Growler  (EA-18G),  and  Super  Hornet  (F/A-18E/F).  Each  of  these 
programs  demonstrated  that  an  integrated  government/contractor  teaming  arrangement 
for  Test  and  Evaluation  (T&E)  provided  economies  in  cost  and  schedule  as  well  as 
enhanced  overall  performance  of  the  T&E  effort.  Leveraging  this  philosophy  and 
applying  lessons  learned,  the  P-8A  T&E  approach  is  also  using  an  Integrated  Test  Team, 
whereby  both  the  prime  contractor  and  the  government  share  the  test  team  activities.  The 
P-8A  program  established  a  single  team  with  representatives  from  the  contractor  and  the 
government  developmental  and  operational  test  communities.  The  task  for  this  integrated 
team  is  to  conduct  the  daily,  on-site  test  and  evaluation  activities.  The  ITT  prepares 
integrated  test  plans,  executes  the  integrated  test  program,  collects  data,  and  maintains  a 
common  database.  Utilizing  the  ITT  concept  maximizes  the  efficient  use  of  time, 
resources,  and  work  force  during  all  phases  of  the  P-8A  test  program  except  Initial 
Operational  Test  and  Evaluation  (IOT&E).  Objectives  of  using  the  ITT  concept  are  cited 
from  the  P-8A  Test  and  Evaluation  Master  Plan  (Office  of  Secretary  of  Defense,  2005): 

•  “Provide  opportunities  for  Developmental  and  Operational  Test  evaluation  of 
the  P-8A  system  early  in  the  development  program.” 

•  “Enable  testing  to  address  technical  and  operational  test  concerns  at  the 
earliest  opportunity.” 

•  “Avoid  duplication  of  government  and  contractor  testing  and  test  data 
requirements.  Combined  testing  provides  an  opportunity  for  government 
testing  to  “piggyback”  on  contractor  development  test  and  thereby  avoids 
extra  flight  hours  to  collect  data  on  measures  that  are  only  of  interest  to  the 
government.” 

•  “Provide  an  environment  for  coordination  of  testing  within  which  ITT 
members  can  effectively  and  efficiently  obtain  data  to  execute  their  combined 
DT/OT  mission.” 
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A  crucial  factor  that  enables  an  effective  and  efficient  ITT  is  the  ability  to  conduct 
day-to-day  business  in  the  most  autonomous  manner  possible.  Authority  is  delegated  to 
appropriate  government  and  contractor  personnel  within  the  ITT  to  the  maximum  extent 
possible.  Operating  in  this  manner  requires  ground  rules  and  clearly  defined  roles  and 
responsibilities  be  assigned.  NAVAIR  and  Boeing  jointly  developed  a  Memorandum  of 
Agreement  (MOA)  that  clearly  identified  roles  and  responsibilities  within  the  test 
program.  This  was  a  lesson  learned  from  the  F/A-18  E/F  program,  which  in  their  report 
back  to  Naval  Air  Systems  Command  leadership  stated,  “The  complex  nature  of  the  ITT 
requires  clear  guidance  of  roles,  responsibilities,  and  conflict  resolution  in  order  for  it  to 
be  successful.”  The  MOA  clarified  issues  such  as  governing  documents,  test  scheduling, 
test  air  space,  test  facilities,  organizational  management,  deficiency  reporting,  and  access 
to  Boeing  and  NAWCAD  Patuxent  River  facilities.  The  MOA  is  a  living  document  and 
is  periodically  reviewed  and  updated  to  reflect  current  programmatic  and  test 
environment. 

There  are  three  main  components  of  the  Integrated  Test  Team.  These  are 
Contractor  Test,  USN  Developmental  Test,  and  Operational  Test.  During  the  SDD 
phase,  components  have  objectives  described  below  and  illustrated  in  Figure  3: 
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Figure  3.  P-8A  Integrated  Test  Team 

•  Contractor  Test  (CT)  objective:  Ground  and  flight-test  SDD  program  to 
develop  the  P-8A  Poseidon  weapon  system  capability  and  to  verify  by  test  and 
demonstration  that  the  aircraft  and  its  systems  meet  contractual  requirements 
as  stated  in  the  Performance  Based  System  Specification  (PBSS). 

•  Developmental  Test  (DT)  objective:  USN  testing  in  conjunction  with 
contractor  testing  to  evaluate  and  verify  capabilities  required  by  the  PBSS,  as 
well  as  evaluation  of  the  weapon  system  capability  from  a  mission  relation 
perspective. 
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•  Operational  Test  (OT)  objective:  This  is  USN  testing  done  in  conjunction 
with  contractor  and  USN  developmental  testing  to  evaluate  mission  suitability 
and  mission  effectiveness  of  the  P-8A  Poseidon  weapon  system  as  developed 
and  evaluated  during  the  ground  and  flight-test  program. 

Testing  is  conducted  at  two  primary  sites.  The  majority  of  the  ground  testing  is 
conducted  at  Boeing  facilities  in  Seattle,  Washington.  This  testing  includes  static  and 
fatigue  testing.  The  primary  site  for  the  flight-test  program  is  the  U.S.  Naval  Air  Warfare 
Center,  Aircraft  Division  (NAWCAD),  Patuxent  River,  Maryland. 

D.  P-8A  TEST  WORKING  DESCRIPTION  (TWD)  PROCESS 

A  test  working  description  (TWD)  is  a  detailed  test  plan  NAVAIR  uses  when 
conducting  a  test.  There  is  a  lengthy  formal  review  and  approval  process  familiar  to  the 
NAVAIR  test  community.  There  are  72  TWDs  required  to  conduct  the  P-8A  verification 
phase.  After  these  TWDs  are  developed,  they  must  undergo  the  review  and  approval 
process.  A  TWD’s  complexity  can  vary  depending  on  the  content  of  the  testing.  The 
level  of  review  and  approval  is  set  by  the  level  of  complexity.  Therefore,  the  complexity 
heavily  influences  the  time  it  takes  to  get  the  TWD  through  the  process.  Figure  4 
illustrates  the  original  TWD  approval  process  that  the  government  test  team  intended  to 
use.  It  also  identifies  the  number  of  reviewers  and  approvers.  The  issue  that  was 
recognized  but  not  thoroughly  understood  by  both  Boeing  and  NAVAIR,  was  the 
complexity  of  adding  Boeing’s  contribution  to  that  process.  Early  attempts  to  map  out 
the  process  and  time  associated  with  this  process  were  completed  at  a  high  level. 
Unfortunately,  these  attempts  lacked  details  to  understand  the  true  impact  on  throughput. 
Figure  5  provides  the  simplistic  view  of  the  TWD  process  and  its  notional  timeline.  The 
NAVAIR  reviewers,  who  would  normally  generate  comments  and  recommendations, 
originally  scoped  the  process  to  have  an  initial  draft  of  the  TWD  followed  by  a  review. 
The  TWD  moves  through  the  process  undergoing  reviews  and  updates  to  mature  the 
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TWD  for  submission  for  approval.  The  final  review  is  the  Executive  Review  Board 
(ERB).  The  ERB  would  either  approve  the  TWD  or  assign  actions  to  be  completed  prior 
to  the  TWD  returning  to  the  ERB. 
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Figure  5.  Early  TWD  Process  Notional  Timeline 

This  notional  timeline  gave  a  false  sense  of  security  that  there  was  sufficient 
understanding  of  process  duration.  This  proved  to  be  in  error  as  the  team  initiated  TWDs 
for  ground  tests.  By  the  time  the  P-8A  program  started  to  understand  the  ramifications  of 
underestimating  the  TWD  process,  the  Boeing  technical  team  was  downsized  in  line  with 
the  program’s  plan  to  maintain  the  program’s  expenditure  profile.  Boeing’s  787  and  747 
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programs  quickly  absorbed  the  personnel  slated  to  be  part  of  the  downsizing.  This 
presented  a  significant  challenge  to  the  P-8A  program  because,  appropriately,  staffing  the 
TWD  process  was  becoming  extremely  difficult  and  the  TWD  backlog  began  to  grow. 

After  several  reviews,  it  became  evident  that  a  key  element  of  maintaining  critical 
program  milestones  was  to  perform  an  assessment  of  the  TWD  process,  seeking 
opportunities  to  identify  possible  bottlenecks  in  the  process  and  detennine  the  appropriate 
staffing  and  resources  required  to  efficiently  develop,  review,  and  approve  the  72  TWDs. 
In  November  2008,  the  P-8A  program  office  and  the  ITT  requested  an  in-depth  review  of 
the  TWD  process  to  include  documenting  a  detailed  process  flow. 

Figure  6  presents  the  detailed  process  flow  that  added  granularity  to  the  process 
and  necessary  insight  to  assess  possible  choke  points.  This  process  included  several 
decision  points  a  TWD  must  pass  through  prior  to  achieving  ERB  approval.  Underneath 
each  of  the  process  blocks  is  the  work  and  coordination  that  takes  place  to  position  the 
TWD  to  pass  each  checkpoint  successfully.  Unaccounted  in  this  detailed  flow  under  the 
original  view,  shown  in  Figure  5,  was  an  increase  in  process  time.  The  added  detail  began 
to  pinpoint  choke  points  within  the  process  as  well  as  highlight  staffing  deficiencies.  The 
detailed  process  also  identified  that  all  TWDs  are  not  the  same  and  groups  them  into 
three  different  categories  (A,  B,  or  C).  Each  category  has  different  approval  paths. 
Category  A  has  the  lowest  complexity  level  with  approval  authority  at  a  lower  level  than 
category  B  or  C.  Category  C  is  the  highest  complexity  level,  takes  the  longest  processing 
time,  and  requires  approval  of  senior  Navy  test  leadership.  Category  B  falls  in  between 
A  and  C  with  respect  to  complexity,  duration,  and  approval  authority.  Even  with  the 
additional  level  of  detail,  this  view  of  the  TWD  process  still  handicapped  the  P-8A 
program  from  fully  understanding  the  risk  the  program  was  exposed  to  in  the  test 
program.  To  help  fully  understand  the  risk,  it  was  decided  to  create  a  computer  model  to 
better  characterize  the  TWD  process  for  optimization  opportunities. 
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E.  TWD  MANAGEMENT  TOOLS 

Many  tools  have  been  used  to  improve  the  efficiency  of  business  processes. 
Some  of  these  are  applicable  to  the  TWD  process.  Two  specific  tools,  Excel  spreadsheets 
and  Business  Process  Reengineering  (BPR),  were  used  earlier  in  the  P-8A  program  with 
the  goal  to  improve  the  efficiency  of  the  TWD  process.  This  section  presents  a 
discussion  of  these  tools. 

1.  Early  TWD  Management  Methods  using  Excel  Spreadsheets 

Early  methods  of  managing  TWDs  through  the  TWD  process  ranged  from  the  use 
of  pencil  and  paper  to  using  Excel  spreadsheets.  Tables  1  and  2  show  samples  of  P-8A 
spreadsheets  used  within  the  ITT.  As  shown  in  Table  1,  the  spreadsheet  tracks  the  name 
of  the  TWD  as  well  as  the  test  team  associated  with  the  TWD.  The  sheet  lists  the  key 
personnel  who  develop  and  usher  the  TWD  through  its  process.  Table  2  tracks  the 
progress  of  each  TWD  against  the  baseline  date.  The  color  codes  for  the  tracker  sheet  are 
‘green’  if  the  TWD  is  on  schedule.  The  yellow  shows  the  TWD  is  off  schedule  but  meets 
need  date  and  red  means  the  TWD  is  off  schedule  and  exceeds  need  date. 


Table  1.  The  P-8A  TWD  Personnel  Resource  Excel  Spreadsheet 


TW)  Title 

Test  Conductor 

ftoject  Engneer 

Project  Officer 

Operational  Test 
[Representative 

Fight  Test 
Brgineer 

Technical 

Lead 

Contractor 
Right  Test 
Director 

Government 
Flight  Test 
Director 

AIRCRAFT  \ABGHr  &  BALANCE 

BTC,  AS 

DT,  FE,  AS,  AERGMECH 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

FUEL  SYSTEM  GRCUNDTEST 

BTC,  AS 

DT,  PE,  AS,  IVECH  StS 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

ECS  GROUND TEST 

BTC,  AS 

DT,  FE,  AS,  MECH  STS 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFID 

ELECTEJ  CAL  S^TBvrt  GROUND  TEST 

BTC,  AS 

DT,  FE,  AS,  IVECH  SYS 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

EMC  SOFT 

BTC,  MS 

DT,  PE,  MS,  EE 

DT,  NFQ  PO,  MS 

OT,  NFO,  FO,  MS 

FTE 

CFTD 

GFTD 

HUMAM  FACTORS 

BTC,  HF 

DT,  PE,  AS,  HUMAN  FACTORS 

DT,  NFQ  PO,  MS 

OT,  l\FO,  FO,  MS 

FTE 

CFTD 

GFTD 

AIRCRB/V  GROUND  EGRESS  DEMO 

BTC,  AS 

DT,  FE,  AS,  MECH  SYS 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

EXTH^NALRBJO  OF  VIEW  GROUND 

BTC,  AS 

DT,  FE,  AS,  GENERC 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

FQFUGHTTEST  -  CLEAN 

BTC,  AS 

DT,  PE,  AS,  FQ 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

STRUCTURAL  LCADS  GROUND& FUGHTTEST  - 

BTC,  AS 

DT,  PE,  AS,  LCADS 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

FLUTTER  FUGKT  TEST  -CLE4N 

BTC,  AS 

DT,  PE,  AS,  FLUTTER 

DT,  PLOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

TAXI/FREUMNAKf  FUQ-fT  B/AL 

BTC,  AS 

DT,  FE,  AS,  FQ 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

HEALTH  MANAOBVBTT  SrSTB/l-  FUGHTTEST 

BTC,  F&M 

DT,  FE,  AS,  MECH  STS 

DT,  PILOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 

PERFORMANCE  FUGHTTEST  -  CLEAN 

BTC,  AS 

DT,  PE,  AS,  FERF 

DT,  PLOT,  PO,  AS 

OT,  PILOT 

FTE 

CFTD 

GFTD 
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Table  2.  The  P-8  A  TWD  Tracking  Spreadsheet 


TWD  TITLE  /  DESCRIPTION 

Aircraft 

TWD  NO. 

Schedule 

TWD 

KICKOFF 

1st 

DRAFT 

Test  Team 

Review 

(TTR) 

Technical  & 
Risk 

Assessment 

<TRA> 

APPROVAL 

(ERB) 

Umbrella  Test  Plan  (per  NAVAIRINST  3960.4B) 

T1.T2.T3 

Umbrella 

Plan 

9/16V08 

1 1/6/08 

11/15/08 

11/15/08 

Forecast 

9/16/08 

11/6/08 

1/8/09 

1/8/09 

Actual 

9/16/08 

11/6/08 

1/8/09 

1/8/09 

Fuel  System  Ground  Test 

T1.T2.T3 

FBA05.01-001 

Plan 

4/1/08 

4/14/08 

6/1/09 

6/22/09 

7/7/09 

Forecact 

4/1/08 

4/14/08 

6/1/09 

6/22/09 

7/7/09 

Actual 

4/1/08 

4/14/08 

Aircraft  Weight  &  Balance 

T1.T2.T3 

FBA1 4.07-001 

Plan 

4/1/08 

9/30/08 

11/19/08 

11/19/08 

12/19/08 

Forecast 

4/1/08 

9/30/08 

2/19/09 

2/19/09 

3/6/09 

Actual 

4/1/08 

9/30/08 

2/19/09 

2/19/09 

3/31/09 

Electrical  System  Ground  Test 

T1.T2.T3 

FBA05. 04-001 

Plan 

4/15/08 

4/28/08 

8/29/08 

8/29/08 

9/29/08 

Forecast 

4/15/08 

4/28/08 

2/13/09 

2/13/09 

3/10/09 

Actual 

4/15/08 

4/28/08 

2/18/09 

2/18/09 

3/10/09 

EMC  SOFT 

T1 

FBA04. 12-001 

Plan 

1/5/09 

5/8/09 

5/22/09 

6/5/09 

6/19/09 

Forecast 

2/23/09 

5/8/09 

5/22/09 

6/5/09 

6/19/09 

Actual 

2/27/09 

Aircrew  Ground  Egress  Demonstration 

T1 

FBA1 4.01 -004 

Plan 

3/2/09 

4/10/09 

5/26/09 

6/9/09 

6/23/09 

Forecast 

3/20/09 

4/10/09 

5/26/09 

6/9/09 

6/23/09 

Actual 

4/2/09 

4/10/09 

Human  Factors  Demonstration 

T2,T3 

FBA1 4.01  -002 

Plan 

2/16/09 

3/27/09 

5/26/09 

6/9/09 

6/23/09 

Forecast 

3/13/09 

5/1/09 

5/26/09 

6/9/09 

6/23/09 

Actual 

Test 

Conductor 

Dean  Armstrong 

Dean  Armstrong 
Dean  Armstrong 
Dean  Armstrong 

Don  Arbuthknot 

Nick  Shields 

Nick  Shields 


Completed 

On  plan/target  for  performance  period 


2.  Business  Process  Reengineering  (BPR)  Application 

Deadlines  for  TWDs  were  quickly  approaching  and  progress  was  minimal.  The 
flight  test  schedule  was  in  jeopardy.  It  became  apparent  that  an  overhaul  of  the  process 
was  necessary  and  that  the  business  process  reengineering  (BPR)  approach  could  apply. 
A  study  from  Hammer  and  Champy  (1993,  cited  in  Nahmias,  2005)  defines  BPR  as  a 
process  of  challenging  the  way  business  is  currently  conducted.  It  does  not  accept  the 
answer,  “because  that’s  the  way  we  do  it.”  Applying  BPR  is  not  very  difficult  but  does 
have  five  general  principles  to  follow.  The  five  principles  are  listed  below,  with  notes  on 
how  they  were  applied  within  the  P-8A  process: 

1.  Several  jobs  are  combined  into  one:  Merging  the  contractor’s  process  with 
Navy  DT/OT  test  plan  process  provided  an  opportunity  to  streamline  and 
integrate  each  process.  The  contractor’s  ability  to  execute  the  test 
program  can  be  combined  with  the  Navy’s  ability  to  conduct  test  programs 
with  ordnance  while  maintaining  a  strong  safety  emphasis. 

2.  Workers  make  decisions:  The  engineers,  test  conductors,  project  officers, 
and  technicians  establish  the  TWD  content,  data  requirements,  success 
criteria, 
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and  test  sequences.  The  engineers  define  what  data  must  clear  a  test 
sequence  while  the  test  conductors  and  technicians  define  the  efficient 
approach  of  obtaining  the  data. 

3.  The  steps  in  the  process  are  performed  in  a  natural  order:  There  were 
initial  struggles  in  this  area  as  the  contractor  and  Navy  began  to  combine 
their  processes  into  a  single  approach.  The  approval  of  a  single  TWD  was 
different  between  the  two  entities.  The  Navy  had  a  much  more  rigorous 
safety  assessment  than  the  contractor.  The  newly  integrated  process 
addressed  results  of  hazard  analysis  at  each  major  process  review. 

4.  Process  should  have  multiple  versions:  The  development  and  approval  of  a 
TWD  employed  two  basic  approaches. 

a.  A  single  writer  of  the  TWD  owns  its  contents  and  its  success 
criteria  and  is  responsible  for  its  final  approval.  The  writer 
consults  with  other  key  technical  members  as  needed. 

b.  A  team  is  assembled  to  oversee  the  TWD’s  creation  and 
approval.  The  team  is  comprised  of  engineering  and  test 
disciplines  that  directly  influence  the  TWD.  A  test  conductor  is 
assigned  as  the  leader  of  the  team.  His  primary  responsibility  is 
ensuring  identification  of  all  key  players,  scheduling  of  the 
approval  board,  and  resolving  team  conflicts.  This  approach 
proved  to  be  effective  in  dealing  with  complex  TWDs. 

5.  Perform  work  where  it  makes  sense:  This  principle  turned  out  to  be  difficult 
with  one  part  of  the  test  team  working  from  the  West  coast  and  the  other  on 
the  East  coast.  They  established  core  hours  to  minimize  inconveniences. 
Leveraging  web  conferencing  allowed  the  TWDs  to  be  worked 
simultaneously  with  both  coasts. 

Employing  the  BPR  technique  generated  areas  of  opportunity  to  improve  the 
process  at  the  macro  level.  The  BPR  identified  duplication  of  effort  during  the  reviewing 
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cycles  of  the  TWD.  The  P-8  A  test  team  leaders  changed  the  reviews  from  a  serial  flow  to 
a  parallel  flow.  The  assigned  team  conducted  Web-based  conferencing  review  enabling 
real-time  editing  of  the  TWD.  These  changes  provided  significant  reduction  in  process 
time  by  obtaining  all  team  members’  participation  and  team  ownership  of  the  TWD  early 
in  the  process.  This  technique  also  allowed  both  the  Navy  and  the  contractor  to  insert 
their  unique  attributes,  test  safety,  and  test  efficiency,  in  the  TWD  up  front.  Despite  the 
improvements  brought  about  by  employing  BPR,  the  P-8A  program  still  lacked  the 
capability  to  predict  TWD  production  rates  as  a  function  of  resource  levels.  The 
shortcoming  in  the  TWD  production  rate  prediction  is  what  led  to  the  author’s  modeling 
efforts  described  in  Chapter  IV. 

F.  CHAPTER  SUMMARY 

This  chapter  provided  an  overview  of  the  P-8A  program.  The  P-8A  is  the 
replacement  of  P-3C  aircraft  having  the  anti-submarine  warfare  (ASW)  as  its  primary 
mission.  The  chapter  provided  the  organizational  structure  of  the  P-8A  program.  It 
described  the  ITT  concept  being  employed  to  conduct  the  verification  phase  of  the 
program.  The  P-8A  program  adopted  lessons  learned  from  other  large  NAVAIR 
AC  AT- ID  programs  such  as  F-18E/F  and  E-2D.  The  chapter  described  the  three  ITT 
objectives,  illustrated  the  ITT  organization,  and  defined  the  different  components  of  the 
ITT. 

Finally,  this  chapter  provided  the  description  of  the  TWD  process  as  it  was 
originally  defined.  It  described  the  shortfalls  and  challenges  of  using  the  original 
simplistic  process  flow.  The  chapter  summarized  the  program  office’s  and  the  ITT’s 
earlier  attempts  to  revise  the  TWD  process,  which  enhanced  its  detail.  This  added  detail 
highlighted  possible  opportunities  to  improve  the  process  but  also  revealed  the  risk 
created  by  not  having  a  full  understanding  of  the  process  characteristics.  At  that  time,  the 
author  initiated  the  TWD  process  modeling. 
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III.  RESEARCH  ANALYSIS 


A.  INTRODUCTION 

This  chapter  presents  the  Department  of  Defense  Directive  (DoD)  5000.01  view 
of  integrated  testing.  It  describes  the  planning  requirements,  the  role  of  the  prime 
contractor,  and  responsibilities  of  developmental  and  operational  testing  personnel.  This 
chapter  also  discusses  lessons  learned  from  F/A-18E/F,  E-2D,  and  V-22  acquisition 
programs,  which  employed  an  integrated  testing  approach.  The  challenges  and  successes 
these  programs  experienced  helped  shape  the  P-8A  integrated  test  program.  This  section 
also  discusses  how  each  of  these  teams  established  their  team  structure  and  how  their 
procedures  benefited  their  programs.  The  focus  of  this  chapter  extends  beyond  the  TWD 
processes  of  earlier  programs;  however,  the  extension  is  needed  because  understanding 
test  execution  is  essential  for  developing  realistic  and  effective  TWDs. 

B.  DEPARTMENT  OF  DEFENSE  POSITION  ON  INTEGRATED  TEST 

DoD  Directive  5000.01  states  that  integrated  test  and  evaluation  needs  to  be 
conducted  throughout  the  acquisition  process.  The  purpose  of  test  and  evaluation  is  to 
assess  the  technical  maturity,  interoperability,  and  operationally  effectiveness,  and 
confirm  compliance  to  technical  perfonnance  requirements.  The  DoD  Instruction 
5000.02  states: 

Developmental  and  operational  test  activities  shall  be  integrated  and 
seamless  throughout  the  phase.  Evaluations  shall  take  into  account  all 
available  and  relevant  data  and  information  from  contractor  and 
government  sources. 

An  Office  of  the  Secretary  of  Defense  Memorandum  issued  in  2007  states: 

To  maximize  the  efficiency  of  the  T&E  process  and  more  effectively 
integrate  developmental  and  operational  T&E,  evaluations  shall  take  into 
account  all  available  and  relevant  data  and  information  from  contractor 
and  government  sources. 
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This  policy,  along  with  that  in  DoD  Directive  5000.01,  empowers  the  T&E 
community  to  achieve  the  goal  of  early  identification  of  technical,  operational  and  system 
deficiencies  so  that  corrective  actions  can  be  incorporated  in  a  timely  fashion. 

C.  REVIEW  OF  OTHER  PROGRAMS  THAT  USED  INTEGRATED  TEST 

TEAM  APPROACH 

Employing  an  integrated  test  approach  is  not  a  new  technique  within  the  DoD  and 
especially  within  the  USN.  The  United  States  Air  Force  (USAF)  pioneered  integrating 
contractor  and  government  testing  and  modeled  it  after  the  Integrated  Product  Team 
(IPT)  found  throughout  government  and  industry.  VanderVliet  and  Price  (1996)  identify 
the  USAF  as  successfully  employing  combined  test  team  approach  in  1972  on  the  F-16 
program,  the  first  Acquisition  Category  1  (ACAT  1)  program.  The  USN  slowly  began  to 
adopt  this  new  philosophy  within  its  T&E  community.  Several  aircraft  test  programs 
successfully  integrated  developmental  and  operational  testing.  An  integrated  test  team 
comprised  of  government  developmental  and  operational  testers  and  prime  contractor 
testers  was  employed  to  conduct  testing.  Their  success  did  not  come  without  overcoming 
several  hurdles  that  developed  into  lessons  learned  for  other  programs’  application.  From 
these  lessons  learned,  the  P-8  A  Integrated  Test  Team  established  their  operating 
procedures,  roles  and  responsibilities,  and  team  structure  on  expectation  of  achieving 
similar  efficiencies. 

1.  V-22  Integrated  Test  Team  Lessons  Learned 

The  USN  embarked  on  the  road  to  integrated  test  in  February  of  1993  with  the 
V-22  Osprey  program.  The  goal  was  to  reduce  redundant  flight-testing  between 
contractor  testing  and  government  testing  that  was  the  normal  mode  of  verification 
testing  at  this  time.  This  would  allow  for  early  detection  of  design  deficiencies  and 
corrective  actions  without  adversely  impacting  the  program  schedule  and  cost.  Early  in 
the  test  effort,  the  V-22  program  discovered  contractual  roadblocks  that  limited 
government  pilots  and  engineers  to  acting  only  as  monitors.  The  limited  scope  of 
government  test  participation  prevented  the  achievement  of  government  developmental 
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test  objectives.  This  resulted  in  significant  lost  opportunity  of  efficiency.  Because  of  the 
limited  participation,  the  government  had  to  conduct  its  own  separate  developmental 
flight-test. 


Figure  7.  V-22  Test  Asset  at  NAWCAD  Pax  River 

The  program  continued  to  face  challenges  as  it  entered  into  new  territory  within 
the  USN  test  community.  One  specific  area  of  contention  was  that  the  combined  testing 
with  the  contractor  threatened  the  government  test  team’s  independence  and  authority. 
Ultimately,  the  defining  of  the  roles  and  responsibilities  in  a  Memorandum  of  Agreement 
(MOA)  initiated  the  team-building  process.  In  addition,  team  members  recognized  they 
would  need  daily  operational  procedures  that  included  jointly  planned  flight  operations  if 
they  were  to  function  effectively  as  a  team.  These  procedures  became  known  as 
ITTOPS,  Integrated  Test  Team  Operating  Procedures.  The  development  of  these 
procedures  revealed  to  the  team  the  scope  of  work  in  front  of  them.  ITTOPS  also  helped 
foster  buy-in  by  all  members  that  proved  valuable  as  the  testing  progressed.  One  other 
element  that  infused  the  drive  to  “succeed  as  an  integrated  test  team”  was  that  both  test 
teams  were  accountable  for  the  execution  of  the  test  program.  This  emphasized  the 
importance  of  communication  of  new  ways  of  conducting  testing  to  infuse  efficiencies 
into  the  test  program. 
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Lessons  learned  for  future  programs  drawn  from  the  V-22  program  were: 

1 .  Ensure  the  contract  is  structured  to  allow  for  maximum  participation  of 
the  government  representatives  to  achieve  optimal  test  efficiency. 

2.  Establish  a  MO  A  early  in  the  program  that  clearly  defines  roles  and 
responsibilities  to  foster  the  team  building  process. 

3.  Develop  and  implement  1TTOPS  that  are  mutually  approved  by  the 
contractor  and  affected  government  entity. 

These  lessons  learned  were  recognized  and  were  leveraged  into  the  P-8A  test 
program. 

2.  F/A-18E/F  Test  Team  Lessons  Learned 

The  F/A-18E/F  program  recognized  early  that  clear  direction  on  the  execution  of 
the  test  program  was  necessary,  since  it  was  moving  away  from  the  traditional  USN  test 
philosophy.  They  realized  that  mutual  agreement  was  required  on  several  key  documents 
to  minimize  confusion,  conflict,  and  to  allow  for  maximum  efficiency  of  the  test 
program.  These  documents  were: 

•  The  contract  that  governed  the  legal  responsibilities 

•  The  Test  and  Evaluation  Master  Plan  (TEMP)  the  government  document 
that  defines  the  acceptable  performance  requirements  for  the  test  program 

•  The  Master  Test  Plan  that  resides  with  the  contractor  and  details  the  entire 
weapons  system  test  program 

•  A  Memorandum  of  Agreement  (MOA)  established  between  the  contractor 
and  government  defining  roles  and  responsibilities  within  the  ITT 
(Springsteen,  Bailey,  Nash,  &  Woolsey,  1999). 
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Figure  8.  F/A-18  Super  Hornet  at  Pax  River 

The  contract  assigned  the  contractor  with  the  responsibility  of  being  the  team  lead 
within  the  ITT.  This  was  a  first  for  USN  testing  and,  as  with  the  V-22  test  program,  the 
government  test  team  viewed  this  as  making  them  subservient  to  contractor.  The  contract 
vehicle  defused  some  of  the  caustic  issues,  but  it  did  not  eliminate  all  of  them. 

The  program  reinforced  the  position  of  an  integrated  test  program  by  capturing 
the  requirements  within  the  F/A-18E/F  TEMP  and  the  Master  Test  Plan.  The  TEMP 
directs  the  structure  and  scope  of  the  developmental  and  operational  test  program  to 
include  scheduling  and  resource  allocation.  It  defines  the  Government  Furnished 
Equipment  (GFE),  test  venues,  required  work  force,  and  necessary  training.  The 
contractor  complement  to  the  TEMP  is  the  Master  Test  Plan.  It  is  maintained  by  the 
contractor  and  provides  details  of  the  entire  weapons  system  developmental  test  program 
from  initial  ground  tests  of  models  and  test  articles  through  the  flight  and  system  tests 
leading  up  to  Operational  Evaluation  (OPEVAL). 

The  final  key  controlling  document  that  was  an  enabler  to  the  integrated  test 
program  was  the  F/A-18E/F  MOA.  The  program  leadership  recognized  the  complexity 
of  integrating  the  contractor  and  government  test  team  and  that  team  roles  and 
responsibilities  would  require  an  agreement  by  all  parties.  The  MOA  clearly  defined 
ground  rules,  test  plan  approval  procedures,  roles  responsibilities,  and  authority.  Quite 
often  during  the  test  program,  the  team  members  referred  to  it.  The  MOA  also  addressed 
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a  key  discrepancy  reporting  process  that  had  large  implications  to  the  contractor.  This 
reporting  process  is  discussed  in  more  detail  later  in  this  section. 

The  program  moved  into  new  territory  by  combining  test  plans.  This  resulted  in  a 
valuable  undertaking.  In  a  traditional  test  program,  the  contractor  would  write  a  test  plan 
and  submit  it  to  the  USN.  The  USN  would  take  the  contractor  test  plan  as  a  model  to 
construct  their  own  test  plans  that  they  would  use  for  evaluation.  It  became  clear  that 
they  needed  a  single  combined  test  plan  format  as  part  of  the  integrated  testing  concept. 
To  accomplish  this,  the  contractor  and  USN  ITT  leadership  created  a  test  plan  format  that 
combined  all  of  the  elements  of  the  contractor  and  USN  test  plans  under  one  cover  and 
with  one  sign-off  and  approval  sheet.  This  turned  out  to  be  difficult  at  the  start,  but  later 
resulted  in  a  great  team-building  process. 

One  additional  key  factor  that  allowed  the  program  to  be  successful  was 
establishing  a  deficiency  reporting  process.  The  process  had  large  implication  to  the 
contractor  because  disagreement  on  a  deficiency  causes  the  test  program  to  stop  until  a 
resolution  was  reached.  The  contractor  was  responsible  for  execution  of  the  test 
program.  They  are  also  responsible  for  the  delayed  schedule,  and  ultimately  the 
increased  cost  of  the  program.  The  USN  test  team  viewed  this  process  as  an  opportunity 
to  display  their  independence.  To  develop  a  process  that  all  parties  could  operate  under 
proved  to  be  challenging  but  the  team  was  successful  in  establishing  one.  The  process 
was  fonnal  and  meticulous  and  instituted  discipline  due  to  the  high  visibility  the 
discrepancies  received  throughout  the  ITT  and  up  through  the  higher  echelons  of 
NAVAIR  and  Boeing.  Figure  9  displays  the  process  that  describes  the  start  of  an 
anomaly  as  a  watch  item  and  the  disposition  actions  that  it  could  take  to  be  resolved.  If 
immediate  corrective  actions  are  effective,  the  watch  item  can  be  resolved.  If  the  watch 
item  is  not  resolved,  it  converts  to  a  white  paper.  Finally,  if  the  issue  is  not  able  to  be 
resolved,  the  white  paper  becomes  a  discrepancy  requiring  redesign  and  retesting.  The 
ITT  needed  to  “disposition”  these  issues  quickly  so  mutual  agreements  on  definitions, 
disposition  board  composition,  and  final  authority  were  critical. 
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Summarizing  lessons  learned  from  the  F/A-18E/F,  the  following  steps  were 
recommended: 

1.  Establish  controlling  documentation  early  to  achieve  buy-in  from  both 
contractor  and  government  testers  as  well  as  program  management 

2.  Combine  test  plan  development  and  approval  to  lean  the  process  and 
leverage  an  opportunity  for  team  building 

3.  Establish  a  discrepancy  reporting  process  that  with  joint  agreement  of 
anomaly  criteria 

3.  E-2D  Integrated  Test  Team  Lessons  Learned 

The  most  recent  Navy  aircraft  program  to  enter  into  the  integrated  test  arena  is  the 
E-2D.  The  program  was  built  on  lessons  learned  from  the  V-22  and  F-18E/F  but  also 
generated  methods  of  few  of  their  own.  The  E-2D  program  established  new  benchmarks 
within  the  Naval  test  community  by  taking  the  integrated  test  to  the  next  level  by 
initiating  the  flight-test  program  in  April  2007  at  Northrop  Grumman’ s  facility,  their  East 
Coast  Manufacturing  Center  in  St.  Augustine,  Florida.  Prior  programs  had  conducted  the 
flight-test  at  Naval  Air  Warfare  Center,  Aircraft  Division,  Patuxent  River,  Maryland, 
which  required  the  contractor  to  position  its  test  team  at  the  government  test  facility.  The 
E-2D  program  decided  to  conduct  the  initial  phase  of  their  flight-testing  at  the  contractor 
facility.  This  required  prepositioning  a  core  government  test  team  at  the  contractor’s 
facility.  This  change  resulted  in  new  challenges  facing  the  ITT  regarding  the  roles  of  the 
organization.  The  contractor  flight-test  director  (CFTD)  was  assigned  the  responsibility, 
authority,  and  accountability  for  the  test  program  but  the  government  flight-test  director 
(GFTD)  was  delegated  with  the  final  authority  over  the  daily  flight  schedule.  This 
conflict  of  authority  often  influenced  the  schedule  when  not  resolved  quickly.  NAVAIR 
gave  CFTD  the  authority  over  the  daily  flight  schedule,  and  this  solved  the  problem. 
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Figure  10.  E-2D  Advance  Hawkeye  Test  Asset 

Another  feature  resulting  from  the  government  prepositioning  their  core  team  at 
the  contractor  facility  was  that  the  government  team  was  completely  comprised  of 
volunteers.  The  additional  insight  that  the  E-2D  ITT  participants  discovered  was  the  core 
team  members  needed  to  be  mission-oriented,  open-minded,  and  cooperative  to 
compensate  for  the  reduced  team  size. 

The  working  rules  were  also  different  for  the  contractor  and  government 
personnel.  Each  group  had  different  holiday  schedules,  overtime  rules,  and  award 
compensation  practices.  While  not  desirable,  these  differences  were  largely  unavoidable 
under  the  strict  government  guidelines  and  work  rules.  Establishing  a  common  set  of 
work  rules  became  necessary  to  maintain  morale  in  this  case  where  the  government  and 
contractor  test  teams  worked  in  the  same  facilities. 

Key  lessons  learned  from  the  E-2D  ITT  included  the  following  guidelines: 

•  Provide  the  necessary  authority  commensurate  with  the  assigned 
accountability. 

•  Assemble  the  core  team  that  is  focused  on  positioning  the  test  program  for 
success. 

•  Create  an  equal  working  environment  for  all  members  of  the  team. 
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D. 


CHAPTER  SUMMARY 


This  chapter  examined  the  DoD  perspective  on  integrated  testing,  beginning  with 
the  guidance  in  the  formal  acquisition  instructions.  It  also  researched  a  few  of  the  larger 
USN  aviation  programs  that  have  applied  these  instructions.  While  these  programs 
discovered  challenges,  they  were  able  to  resolve  those  issues  and  to  execute  a  successful 
integrated  test  program.  Each  program  developed  lessons  learned  for  later  programs  to 
leverage  into  their  own  program.  A  common  theme  that  each  team  conveyed  is  to 
identify  roles  and  responsibilities  that  included  accountability  and  authority  early  in  the 
program.  The  ability  for  government  and  contractor  to  operate  as  an  integrated  team 
takes  commitment  and  good  communications.  Combining  processes  to  reduce  the 
duration  of  the  testing  has  tremendous  advantages  but  requires  both  sides  to  yield  on 
some  their  long  held  traditions.  The  P-8A  program  built  their  ITT  from  the  lessons 
learned  by  the  programs  that  preceded  them  and  has  successfully  incorporated  most  of 
those  lessons.  The  P-8  A  program  also  recognized  that  an  optimized,  combined  TWD 
process  was  a  key  to  successful  testing  and  that  there  was  an  opportunity  to  pass  on  an 
effective  lesson  learned  to  following  programs. 


28 


IV.  TEST  WORK  DESCRIPTION  MODELING 


A.  INTRODUCTION 

This  chapter  examines  the  various  software  models  assessed  for  use  in  building 
the  TWD  model.  The  chapter  discusses  the  criteria  applied  in  selecting  the  software  and 
presents  the  chosen  software.  This  chapter  also  looks  at  the  construction  of  the  TWD 
model.  It  presents  the  assumptions  and  inputs  used  in  the  model.  Finally,  this  chapter 
discusses  the  different  concepts  analyzed  by  the  model  and  presents  early  results  from  the 
model. 


1.  TWD  Model  Process 

To  detennine  opportunities  to  improve  the  efficiency  of  the  TWD  process,  it  was 
necessary  to  model  the  TWD  process,  as  understood,  in  its  unmodified  state.  To  build  the 
original  model,  the  author  began  by  interviewing  several  members  of  the  ITT 
representing  both  government  DT/OT  and  the  contractor  membership  and  revisiting  the 
detailed  process  shown  in  Figure  6  and  repeated  in  Figure  11.  The  members  used  the 
detailed  process  displayed  in  Figure  1 1  as  the  baseline  functional  description.  It  was 
interesting  to  note  that  while  there  was  a  consistent  understanding  among  the  members  at 
the  macro  level,  represented  in  the  figure,  the  viewpoints  began  to  diverge  while  mapping 
the  process  details. 
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The  re-examination  of  the  TWD  process  uncovered  two  opportunities  for 
improvement.  These  improvements  resulted  in  the  modification  of  the  baseline  process 
from  a  five-phase  approach  to  an  improved  four-phase  approach  and  streamlined  the 
approval  process  for  TWDs  that  were  lower  in  complexity.  The  initial  improvement 
modified  the  test  plan  working  group  sub-process  to  capture  the  first  draft  of  the  TWD 
and  included  developing  an  entry  criteria  checklist  for  each  technical  review.  Under  the 
modified  process,  the  TWD  entered  the  TPWG  phase  with  mature  engineering  and  test 
requirements,  before  moving  to  the  next  phase.  The  draft  remained  with  the  TPWG  until 
the  assigned  test  conductor  determined  its  maturity  for  submission  to  the  TTRB.  In 
conjunction  with  this  new  four-phase  approach,  definitions  of  entry  and  exit  criteria  for 
the  TTRB  and  ERB  were  established.  This  approach  is  consistent  with  the  NAVAIR 
Systems  Engineering  Technical  Review  (SETR)  process,  which  has  been  a  valuable  tool 
for  program  management.  Figure  12  shows  the  improved  TWD  process.  The  four-phase 
approach  consists  of  requirements  review,  first  draft,  TRA,  and  ERB.  The  final  product 
from  the  ERB  is  an  approved  TWD. 

The  first  phase  of  the  improved  process  is  the  requirement  phase  of  the  TWD.  In 
this  phase,  the  test  conductor  is  responsible  for  assembling  the  requirements  and 
developing  test  content  with  the  assigned  test  team.  The  product  from  this  phase  is  an 
initial  draft  of  the  TWD.  Duration  of  this  phase  can  vary  depending  on  the  complexity  of 
the  TWD  but  the  average  time  is  approximately  five  months. 

The  initial  draft  enters  the  second  phase  where  it  undergoes  a  series  of  team 
reviews  beginning  with  the  TPWG  and  completing  with  TTRB.  These  reviews  prepare 
the  TWD  to  enter  the  fonnal  independent  technical  reviews  by  review  panels  made  up  of 
senior  members  of  the  NAVAIR  test  community. 

The  third  and  fourth  phases  are  the  fonnal  reviews.  The  third  phase  is  the 
technical  readiness  assessment  (TRA),  which  reviews  the  TWD  for  technical  execution, 
safety,  proper  documentation,  and  aircraft  configuration  to  among  other  areas. 
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Figure  12.  Improved  TWD  Process 
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The  fourth  and  final  phase  is  the  Executive  Review  Board  (ERB).  The  challenge 
with  the  ERB  is  availability  of  a  single  individual  who  has  approval  authority.  Some  of 
the  areas  on  which  the  ERB  concentrates  are  personnel  certification,  flight  clearance, 
safety,  test  content,  and  aircraft  configuration.  The  improved  TWD  process  identified  the 
ERB  as  a  potential  choke  point  for  approving  TWDs.  To  help  improve  the  flow  of 
TWDs,  the  Test  community  increased  the  number  of  approving  officials  from  one  to 
three. 

The  second  improvement  opportunity  came  as  it  became  clear  that  not  all  TWDs 
are  the  same  but  rather  fall  into  three  categories.  The  three  categories  as  defined  by 
NAVAIR  Instruction  3960.4B  are  category  A,  B  and  C,  based  on  complexity  and  type  of 
testing.  Category  A  is  the  least  complex  and  requires  the  minimum  amount  of  time  to 
process  for  approval.  An  example  of  category  A  is  antenna  pattern  testing  or  laboratory 
testing.  The  most  complex  TWD  type  is  category  C.  It  requires  a  greater  amount  of  time 
and  resources  than  the  other  two  categories.  An  example  of  category  C  is  flutter  testing. 
Correctly  identifying  the  correct  category  streamlined  the  approval  process  for  the 
category  A  TWDs  by  allowing  a  lower  level  authority  to  approve  the  TWD. 

The  above  process  changes  did  increase  the  quality  of  TWD  write-ups.  Time 
savings  may  also  appear  as  the  ITT  becomes  more  familiar  with  the  process  changes. 

2.  Commercially  Available  Software  Research 

Based  on  review  of  the  initial  functional  decomposition,  the  TWD  process 
improvements  were  still  not  sufficient  to  complete  the  TWDs  in  time  to  support  the  flight 
test  schedule.  In  addition,  the  ability  to  identify  required  resources,  analyze  different 
scenarios,  project  completion  dates,  and  automate  tracking  for  each  TWD  was  still 
missing.  The  manual  tracking  of  the  TWDs  was  cumbersome,  time  consuming,  and 
prone  to  errors.  The  program  needed  an  automated  system. 

Initial  efforts  to  develop  a  unique  software  suite  to  model  the  TWD  process  were 

abandoned  because  of  the  program’s  urgency  to  have  an  operating  system  before 

upcoming  T-l  Test  Readiness  Review  (TRR)  scheduled  for  April  2009.  The  ITT 
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commenced  pursuit  of  more  readily  available  commercial  modeling  software  that  would 
allow  meeting  the  TWD  process  schedule  requirement.  Commercially  available  process 
modeling  software  is  prevalent  and  most  are  capable  of  simulating  all  of  the  desired 
functions.  The  following  are  names  of  the  software  programs  that  were  assessed:  Simcad 
Pro  (Dynamic  Simulation,  2009),  Savvion  Process  Modeler  (Savvion  Process  Modeler, 
2009),  Corel  iGrafx  (iGrafx  Process  for  Six  Sigma,  2009),  and  Rockwell  Automation 
Arena  (Value  with  Enterprise  Wide  Simulation-Corporate  Perspective,  2009).  Each 
software  suite  had  very  similar  capabilities.  These  capabilities  include: 

•  conducting  simulations 

•  generating  tabular  reports 

•  graphical  output 

•  performing  risk  and  statistical  analysis 

•  conducting  what-if  drills 

•  creating  process  maps  and  lean  value  stream  map  diagrams 

•  analyzing  schedule  for  improvement  opportunities 

•  assessing  resources  requirements 

These  business  processing  modeling  software  packages  support  a  broad  range  of 
applications  throughout  today’s  industries.  Health,  energy,  clothing,  trucking,  and 
manufacturing  industries  all  use  the  software.  Businesses  are  continuously  looking  to 
reduce  the  cost  of  doing  business  and  applying  this  software  allows  them  to  optimize 
their  business  processes  (Dynamic  Simulation,  2009).  Most  of  the  assessed  software 
suites  have  Six  Sigma  tools  that  provide  a  capability  to  create  value  stream  maps.  Value 
stream  mapping  allows  for  the  analysis  of  the  flow  of  material  and  information  leads  to  a 
final  product.  The  software  suites  reviewed  had  similar  capabilities  as  far  as  building 
process  maps,  generating  reports,  running  simulations  using  normal  and  uniform 
distributions  to  create  the  demand  on  resources,  and  duration  of  required  resources 
demand. 
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3.  Modeling  Software  Selection 


Based  on  initial  stakeholder  interviews,  requirements  were  developed,  constraints 
documented  and  the  software  suites’  ability  to  meet  those  requirements  was  assessed. 
Among  the  key  requirements  were  the  ability  to  perform  statistical  analysis  and  generate 
graphical  outputs  such  as  value  stream  maps,  compatibility  with  Microsoft  products, 
relatively  low  cost,  and  availability.  Table  3  lists  those  attributes  and  requirements. 


Table  3.  Requirements  for  Process  Optimizing  Software 


Operating  Software 

Be  compatible  with  Microsoft  Windows 

Software  Tools 

Be  capable  of  Statistical  Analysis,  Cost  &  Schedule 
Analysis,  Value  Stream  Mapping,  Flow  Chart,  Risk 
Analysis,  Process  Optimization,  Graphing 

Graphical  Output 

Be  compatible  with  Microsoft  products 

Tabular  Output 

Be  compatible  with  Microsoft  products 

Cost 

Cost  below  $1 ,500 

Availability 

Be  available  within  15  days 

Simulation  &  Modeling 

Have  built-in  capability,  scalable, 

Graphical  User  Interface  (GUI) 

Be  Interactive,  User  Friendly 

Import/Export  Files 

Produce  Microsoft  compatible  files 

Tutorials,  Help 

Must  be  Interactive,  User  Friendly 

Hardware  Requirements 

Must  be  capable  to  be  run  on  Desktop  or  Laptop 
with  160GB  hard  drive,  2  MB  RAM,  1.8MHz 

The  four  software  suites  were  scored  against  the  requirements.  The  author 
employed  a  simple  design  selection  matrix  to  compare  software  candidates.  Software 
meeting  each  requirement  received  a  score  of  five  (5);  a  partially  met  requirement 
received  a  score  of  three  (3);  and  a  software  not  meeting  the  requirement  received  a  score 
of  zero  (0).  The  requirements  were  reviewed  with  the  Government  Flight  Test  Director 
(GFTD)  and  their  priorities  established.  Based  on  their  importance,  requirements  were 
weighted  by  the  GFTD  who  is  one  of  primary  recipient  of  the  of  the  model  output.  The 
author  totaled  scores  for  each  software  suite  evaluated.  Results  from  the  scoring  event 
revealed  Corel  iGrafx  was  marginally  better  than  Rockwell’s  Arena.  A  major  factor  that 
gave  iGrafx  the  edge  over  Arena  was  the  cost  and  availability.  The  Naval  Air  Systems 
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Command  employed  iGrafx  to  support  their  process  improvement  initiatives.  This 
allowed  for  immediate  access  to  the  software  and  eliminated  the  software  cost  to  the 
program.  Table  4  contains  the  results  of  the  assessment. 


Table  4.  Scoring  of  Software  Suites 


Name  Software 

Modelinq  Software 

Attributes 

% 

Weiqhted 

Simcad 

Pro 

Savvion 

Business 

Manager 

7.0 

Corel  iGrafx 
2007 

Rockwell 

Automation 

Arena 

Operating  Software 

10% 

0.5 

0.5 

0.5 

0.5 

Software  Tools 

15% 

0.75 

0.75 

0.75 

0.75 

Graphical  Output 

10% 

0.5 

0.5 

0.5 

0.5 

Tabular  Output 

10% 

0.5 

0.5 

0.5 

0.5 

Cost 

15% 

0 

0 

0.75 

0.45 

Availability 

15% 

0.25 

0.25 

0.25 

0.25 

Simulation  &  Modeling 

15% 

0.75 

0.75 

0.75 

0.75 

Graphical  User  Interface 
(GUI) 

5% 

0.25 

0.25 

0.25 

0.25 

Import/Export  Files 

5% 

0.25 

0.25 

0.25 

0.25 

Total  Score 

100% 

3.75 

3.75 

4.5 

4.2 

Scoring  Criteria:  5  -  Requirement  Met;  3  -  Requirement  Partially  Met;  0  -  Requirement  Not 


Met 


B.  MODEL  DEVELOPMENT 

1.  Model  Objective 

The  objective  of  this  model  is  to  provide  the  capability  to  the  ITT  and  the  P-8  A 
Program  Office  to  explore  the  impact  of  resource  levels  or  throughput  of  test  plans 
(TWDs)  on  the  cost  and  schedule  of  the  flight  test  program.  The  ultimate  objective  is  to 
make  sure  TWDs  are  produced  in  a  cost  efficient  manner  on  a  schedule  that  supports  the 
flight  test  programs’  needs. 

In  researching  the  different  approaches  to  model  the  TWD  process  and  resource 
requirements,  there  was  a  large  amount  of  information  on  application  of  business  process 
modeling.  This  was  no  surprise,  since  most  businesses  now  focus  on  leaning  out  their 
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processes  to  minimize  the  cost  of  doing  business.  An  example  of  this  application  within 
NAVAIR  is  the  AIRSpeed  program.  The  sole  purpose  of  AIRSpeed  is  to  review 
processes  within  NAV AIR’s  charter  and  determine  opportunities  to  be  improved  using 
Six  Sigma,  Theory  of  Constraints  principles,  and  commercial  industries  lean  initiatives. 
Consistent  with  the  NAVAIR  AIRSpeed,  the  same  methodologies  and  toolsets  were 
employed  on  the  TWD  process.  Theory  of  Constraints  was  very  applicable  within  the 
TWD  process.  The  process  has  multiple  constraints  modeled  to  determine  which 
constraints  would  yield  the  highest  return  on  investment  if  optimally  adjusted.  In  the 
below  bullets,  Harden  (2004)  provides  some  insight  into  NAVAIR’ s  motivation  to 
change  the  current  culture  to  improve  corporate  processes. 

•  Enterprise  AIRSpeed  integrates  best  business  practices,  which  includes  Basic 
and  Advanced  Theory  of  Constraints,  Lean  and  Six  Sigma.  The  program 
emphasizes  continuous  process  improvement  to  the  Naval  Aviation  culture. 

•  The  NAVAIR  4.1  Competency,  Systems  Engineering,  has  been  actively 
involved  in  improving  organizational  performance  starting  with  Business 
Process  Reengineering  (BPR)  . . . 

In  addition,  objectives  within  AIRSpeed  were  similar  to  those  established  for 
optimizing  the  TWD  process.  Listed  below  are  extracts  from  a  presentation  given  by 
Moore  (2005)  highlighting  those  objectives: 

•  “AIRSpeed  is  all  about  increasing  Productivity  to  reduce  our  cost  of  doing 
business.” 

•  “AIRSpeed  enables  the  extended  enterprise  to  reduce  the  cost  of  doing 
business  using  a  “System-of-Systems”  approach.” 

The  application  of  process  modeling  is  not  just  limited  to  NAVAIR,  but  has  been 
in  use  in  numerous  industries  and  other  DoD  services.  The  USAF  has  applied  lean 
initiatives  tools  within  their  aircraft  depots  (Christopher  2005).  These  tools  are  similar  to 
those  applied  to  the  TWD  process,  such  as  value  stream  mapping,  lean  enterprise  self- 
assessment  tool,  or  PDMCAT1  that  assess  the  life  cycle  of  the  process  across  the 


1  Developed  by  Rand  Corporation  for  Project  Air  Force  as  a  Depot  Capacity  Assessment  Tool. 
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enterprise  and  provide  understanding  of  key  constraints  influencing  the  process  (.Loredo, 
Pyles,  Snyder,  2007).  Other  industries  such  as  healthcare,  manufacturing,  aerospace, 
transportation,  and  supply  chain  management  employ  these  tools  to  reduce  the  cost  of 
doing  business. 

2.  Building  of  the  Model 

The  model  shown  in  Figure  13  was  started  in  iGrafx  by  creating  the  top-level 
process  first  and  then  adding  the  sub-processes  CAT  A  through  CAT  C  on  Figure  13, 
after  the  top-level  process  was  completed.  In  addition,  the  model  used  factors  derived 
from  historical  data  from  multiple  test  programs  conducted  at  NAWCAD  Pax  River.  The 
factors  included  the  likelihood  of  scheduling  a  technical  review  as  well  as  successfully 
passing  the  review.  The  TWD  development  start  date  was  set  at  7  January  2009. 

The  author  entered  the  top-level  process  into  iGrafx  by  selecting  ‘Process’  on  the 
File  Menu.  The  author  selected  process  fonnat  because  it  had  the  ability  to  represent 
accurately  the  TWD  process,  and  is  the  iGrafx  recommended  modeling  format.  Figure 
14  is  a  snapshot  illustrating  the  selection  of  the  model  fonnat.  The  basic  process  flow 
was  mapped  into  iGrafx  by  selecting  the  appropriate  flowchart  symbol.  The  author  used 
standard  flowchart  symbols  in  iGrafx  decision  points,  process  steps,  connectors  and 
tennination  points,  which  made  the  process  creation  straightforward.  Figure  18  shows 
these  symbols. 
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Figure  13.  Snapshot  of  Selecting  Process  Format 


Figure  14.  Menu  of  Flowchart  Symbols 


Each  step  in  the  process  had  specific  properties  assigned  to  control  its  behavior 
within  the  process.  The  property  elements  used  for  each  step  were  input,  task,  resource, 


39 


outputs,  attributes,  and  last  simulation.  The  first  element,  input  property  established  the 
“queuing  rules”  for  how  the  data  enters  and  exits  the  process  step.  The  TWD  model 
queuing  rule  used  was  “first  in  first  out,”  meaning  all  TWDs’  priorities  were  equal  as 
they  enter  into  the  process  step. 

The  next  element  was  resources.  This  element  defined  the  time  required  to 
complete  the  work  associated  with  the  process  step.  It  allowed  for  a  constant  time,  a 
distribution  of  time,  or  an  expression  to  represent  the  resource  allocation.  The  TWD 
model  used  a  constant  allocation  that  was  dependent  on  the  resource  type  and  TWD 
complexity.  Each  step  had  its  resource  pool  built  to  support  that  process  step.  The  initial 
allocation  for  the  TWD  model  was  completed  in  the  Requirements  Review  step.  For 
example,  if  the  TWD  required  a  test  conductor  and  was  a  category  C  complexity,  the 
constant  value  for  the  test  conductor  would  be  0.6,  or  60  percent,  of  his  time  assigned  to 
that  TWD.  If  the  complexity  was  a  category  A  TWD,  the  constant  value  for  the  test 
conductor  would  be  0.4,  or  40  percent,  of  the  test  conductor  time  allocated  to  the  TWD. 

1.  Category  B  and  C 

60  percent  of  time  per  TWD  per  test  conductor 
40  percent  of  time  per  TWD  per  test  team  member 

2.  Category  A 

40  percent  of  time  per  TWD  per  test  conductor 
30  percent  of  time  per  TWD  per  test  team  member 

Multiple  process  steps  such  as  Requirements  Review,  TTRB  Look  Ahead, 
TTRB,  and  TTRB  Action  Resolution  had  the  resource  pool  created  within  iGrafx.  Listed 
below  are  the  inputs  incorporated  into  the  model  as  constraints  within  the  resource  pool 
for  the  different  TWD  categories.  Figure  15  shows  a  snapshot  of  the  resource  pool  used 
for  Requirements  Review. 
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Figure  15.  Screen  Shot  for  Requirements  Review 


The  next  property  element  is  task.  This  property  element  defined  how  long  the 
process  step  takes  to  complete.  Options  for  this  element  were  a  constant  value,  normal 
distribution,  a  uniform  distribution,  or  an  expression.  The  TWD  model  used  a  normal 
distribution  to  represent  the  following  process  steps:  Requirements  Review,  TPWG  (1st 
Draft),  Prepare  for  TRA,  Prepare  for  ERB.  The  distribution  range  was  derived  from 
historical  data  from  prior  NAVAIR  test  programs.  The  minimum  range  was  one  to  three 
weeks  and  the  maximum  range  was  six  to  fourteen  weeks.  The  remaining  steps  were 
constant  duration  and  varied  from  one  to  three  weeks  depending  on  the  process  step. 

The  final  property  element  used  in  the  TWD  model  was  output.  This  element  was 
used  to  establish  decision  criteria  within  the  process  step.  The  TWD  model  used  the 
decision  criteria  to  represent  the  likelihood  of  a  TWD  entering  into  a  technical  review 
successfully  and  receiving  approval  from  the  review.  The  model  used  historical  data 
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from  prior  test  programs  to  establish  the  probabilities  for  each  review.  The  likelihood 
values  for  a  TWD  successfully  passing  each  technical  review  are  given  below. 

1 .  Technical  Review  Board-50  percent 

2.  Technical  Readiness  Assessment-75  percent 

3.  Executive  Review  Board-50  percent 

Figure  16  shows  an  example  of  the  likelihood  entries  for  the  Technical  Readiness 
Review. 


Figure  16.  Decision  Entries  for  Output  Element  for  Technical  Readiness  Review 

The  next  event  in  the  model-build  process  was  the  creation  and  integration  of  the 
sub-processes.  These  sub-processes  addressed  the  approval  paths  that  TWDs  of  the 
different  categories  follow  through  the  ERB  process  and  are  shown  in  Figure  17.  The 
only  difference  between  the  sub-process  for  Category  A  and  Categories  B  and  C  was  the 
likelihood  of  scheduling  a  review.  Figure  18  displays  the  sub-process  for  Category  A. 
The  factors  used  for  the  potential  of  successfully  scheduling  the  review  are  as  follows: 
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1 .  Technical  Review  Board-85  percent 

2.  Technical  Readiness  Assessment-75  percent 

3.  Executive  Review  Board 

•  Category  A-90  percent 

•  Categories  B/C-50  percent 


Figure  17. 


Display  of  Sub-processes  Integration  at  Executive  Review  Board 
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Figure  18.  Category  A  Sub-process 

The  final  step  in  the  TWD  model  construction  prior  to  running  the  simulation  was 
the  creation  of  the  generator.  The  generator  introduces  the  transaction  to  the  process.  In 
this  model,  a  transaction  is  a  TWD  with  its  associated  attributes.  The  generator  is 
populated  by  importing  an  Excel  file  that  contained  all  the  TWDs  and  their  associated 
attributes  such  as  complexity,  category,  required  personnel,  and  expected  need  date.  Two 
prime  factors  within  the  Excel  file  were  the  priority  and  complexity  of  the  TWD.  The 
ITT  established  priority  based  on  the  date  the  TWD  needed  to  be  approved  to  support  the 
test  schedule.  The  complexity  was  rated  on  a  1  to  10  scale.  They  based  the  complexity 
score  on  multiple  factors  shown  below: 

•  The  complexity  score  increased  as  the  quantity  of  design/performance 
requirements  verified  by  the  TWD  increased. 

•  The  complexity  score  increased  as  the  quantity  of  ground  and/or  flight- test 
hours  increased. 

•  The  test  category  (A/B/C)  influenced  the  complexity  score.  A  category  C  has 
a  higher  complexity  score  than  a  category  A  or  B.  A  category  B  has  a  higher 
score  than  a  Category  A. 

The  generator  menu  allowed  for  selection  of  a  generator  type.  The  different 
generator  choices  were  completion,  demand,  inter-arrival,  and  timetable.  The  generator 
type  the  TWD  model  used  was  a  demand  generator.  This  generator  type  introduced  a 
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transaction  whenever  the  named  resource  (for  example,  test  conductor)  was  available. 
Figure  19  depicts  the  generator  menu  option  selected  to  import  external  data.  The  data 
were  fed  into  the  TWD  model  as  transactions  were  processed  and  resources  became 
available. 


Figure  19.  Snapshot  of  the  Menu  for  the  TWD  Generator 


C.  TWD  PROCESS  SIMULATION 
1.  Simulation  Model  Inputs 

The  initial  step  of  the  model  is  to  import  the  resource  table  shown  in  Table  5. 
This  table  provides  the  available  workforce  capable  of  supporting  the  TWD  process  and 
establishes  the  basis  for  resource  variables  influencing  the  TWD  process.  The  table 
consists  of  the  name  of  the  TWD,  its  category,  key  members  of  the  ITT  responsible  for 
developing  and  managing  the  TWD,  competency  lower  level  technical  readiness 
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assessments  (TRA),  and  the  Executive  Review  Board  (ERB).  The  key  ITT  members  are 
test  conductor  (TC),  primary  project  engineer  (PE),  primary  project  officer  (PO), 
operational  tester  (OT)  representative,  flight  test  engineer  (FTE),  and  instrument 
operation  engineer  (IOE).  The  data  table  also  contains  the  test  category  of  A,  B,  or  C. 
As  previously  stated,  the  test  category  also  reflects  the  complexity  of  the  test  where  a 
category  A  TWD  is  the  least  complex  and  category  C  TWD  is  the  most  complex.  The 
complexity  also  determines  the  number  of  technical  reviews  and  the  time  required  for 
each  ITT  member  supporting  the  TWD  process.  Under  each  of  the  ITT  members  is 
further  specification  of  areas  of  specialization.  For  example,  under  TC  and  radar  TWD, 
the  requirement  is  the  individual  must  be  from  Boeing  (B)  and  have  a  mission  systems 
(MS)  background.  Actual  allocated  support  hours  were  imported  into  the  model.  Figure 
20  provides  a  snapshot  of  the  Government’s  ITT  labor  profile.  The  allocation  of  labor 
becomes  a  constraint  within  the  model. 
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Table  5.  TWD  Process  Personnel  Data 


TWDNbre 

Test 

CbncLdcr 

Reject  Big  iw 

Ftajed 

Qficer 

QErdicrel  Test 
Rpesettcth© 

FicjtTest 

Rgneff 

Tedricd 

Bgreer 

Gcrtradcr 

RicptTest 

Diedcr 

Gbuerrrnnt 

RicjITest 

□rector 

TR^/SLUrrity 

VK-2D  Scatter 
BO 

afcrb™bgtf& 

BALANCE 

BO  ZB 

F5FFEZB 

ZOOvECH 

DFRLOF 

FQZB 

OF  RIOT 

FTE 

ZBTHDH 

LEO 

CFTD 

ODD 

51.63 

CATABB 

FLB-SrSIBVIGOIND 

TEF 

BO  ZB 

F5FFEZB 

IVEDHSrS 

DFRLOF 

FQZB 

OFRLDT 

FTE 

ZBTHDH 

LEO 

CFTD 

ODD 

51.0H- 

IVBD-BrS 

CATBBB 

BSOBLNDTEF 

BTQ  zb 

F5FFEZB 

IVBDHSrS 

DFRLOF 

FQZB 

OF  RIOT 

FTE 

ZBTHDH 

LEAD 

CFTD 

ODD 

51.613 

CATABB 

BHDTRCA-SrSTBVI 
GRIND  lt±>l 

BTQ  zb 

F5FFEZB 

IVBDHSrS 

DFRLOF 

FQZB 

OFRLDT 

FTE 

ZBTHDH 

LEAD 

CFTD 

ODD 

51.9H- 

IVBD-BrS 

CATBBB 

ETVCSBT 

BTQivs 

EDF  FE IVB  h+ 

DFNQFQ 

IVB 

OF  NO  FQ  IVB 

FTE 

IVBTHDH 

LEO 

CFTD 

ODD 

54.45 

CATABB 

FLM'NFZ'CRDB 

STQFF 

EDFFEZB 

HJVPN 

FADIGRS 

DFNQFQ 

IVB 

OF  NO  FQ  IVB 

FTE 

IVBTHDH 

LEO 

CFTD 

ODD 

51.63 

CATABB 

ARDBWCRDJHD 

B3B3SFBVO 

BTQ  ZB 

OFFERS 

IVBDHSrS 

DFRLOF 
PG)  ZB 

OFRLDT 

FTE 

ZBTHDH 

LEO 

CFTD 

CFTD 

51.0H- 

IVBDHSrS 

CATBBB 

BCTETALRBDCFVIB/V 

CRDJSD 

EBVOBTTAflCN 

BTQ  ZB 

F5FFEZB 

GENERC 

DFRLOF 
PD  ZB 

OFRLDT 

FTE 

ZBTHDH 

LEO 

CFTD 

CFTD 

51.63 

CATABB 

FQRJGtnBT  -  CLEAvl 

BTQ  ZB 

DFFEZBFQ 

DFRLOF 
PD  ZB 

OFRLDT 

FTE 

ZBTHDH 

LEO 

CFTD 

CFTD 

51.GH- 

ZOOVBDH 

CATCBB 

STTBCnLFA-LDBS 

(rajSD&RJGTFTEF- 

CLEAM 

BTQ  ZB 

□T,  FEZ'S 
LOB6 

DFRLOF 

FQZB 

OFRLDT 

FTE 

ZBTHDH 

LEO 

CFTD 

CFTD 

51.9H- 

ZBRDvBDH 

CATCBB 

RDnERFUGHTItST  - 

OBN 

BTQ  ZB 

IDT,  FEZ'S 
FLUFTBR 

DFRLOF 

FQZB 

OFRLDT 

FTE 

ZBTHDH 

LEO 

CFTD 

CFTD 

51.6H- 

ZOOVBDH 

CATCBB 
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SDD  TWP  REQUIREMENTS 

TEAM 

SUB 

TEAM 

Name 

RD# 

COST 

CAT 

GS 

Level 

LOC 

Code 

Functional  Area 

WY 

LEAD 

PT 

Lead 

Mac  Brown 

223702,  223699, 
223695 

GL 

14 

PAX 

5.1E 

Product  Test  Team  Lead 

0.90 

PT 

Lead 

Tony  Schmidt  (On-Site) 

223625,  223624, 
223622 

GL 

14 

PAX 

5.1E 

Product  Test  Team  Deputy 

0.90 

PT 

Lead 

Kelly  VanRyswick  (On-Site) 

98817 

CL 

PAX 

CSS-ManTech 

System  Eng/T&E  Analyst 

0.50 

PT 

Lead 

TBD  (VX-1)  (Off-Site) 

Exempt 

CL 

OS 

PAX 

CSS-RBC 

Operational  Test  Analyst 

1.00 

PT 

Lead 

Tanya  Bassett  (On-Site) 

98814 

CL 

PAX 

CSS-RBC/Wyle 

AV  Verification  Matrix 

1.00 

PT 

Lead 

Mark  Sweet  (CDR) 

98811 

GL 

15 

PAX 

5.1G 

Chief  Test  Engineer  (VX-20) 

0.05 

PT 

Lead 

Steve  Gateau  (On-Site) 

98816 

CL 

PAX 

CSS-RBC/Precise 

Information  Technology 

1.00 

PT 

Lead 

Material 

MAT 

PAX 

Material 

PT 

Lead 

T  ravel 

TVL 

PAX 

T  ravel 

ITT  Lead  (GFT 

D) 

PT 

ITT 

TBD  (  On-Site)  (ITT) 

CL 

PAX 

CSS 

GFTD  Admin  Support 

0.00 

PT 

Lead 

Amy  Mattingly  (On-Site) 

Exempt 

CL 

PAX 

CSS  -  RBC 

ITT/T&E  Analyst 

1.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

256398 

CL 

PAX 

CSS 

ITT  Scheduler 

0.25 

PT 

ITT 

TBD  (On-Site)  (ITT) 

CL 

PAX 

CSS 

ITT  Scheduler 

0.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

CL 

PAX 

CSS 

Budget  Support  Analyst 

0.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

CL 

PAX 

CSS 

IT  Coordinator 

0.00 

PT 

Lead 

Troy  Knott  (On-Site) 

Exempt 

CL 

PAX 

CSS-RBC 

Security  (Physical) 

0.00 

PT 

ITT 

Pat  Leard  (On-Site)  (ITT) 

256396 

GL 

14 

PAX 

5.1.3 

Test  and  Data  Lead 

0.80 

PT 

ITT 

TBD  (On-Site)  (ITT) 

256397 

GL 

13 

PAX 

5.1.3 

FCRA 

0.10 

PT 

ITT 

TBD  (On-Site)  (ITT) 

GL 

13 

PAX 

GFR 

0.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

GL 

13 

PAX 

GGFR 

0.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

GL 

12 

PAX 

Photographer 

0.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

GL 

13 

PAX 

ALSS-T2 

0.00 

PT 

ITT 

TBD  (On-Site)  (ITT) 

GL 

13 

PAX 

A/C  Config  Mgmt 

0.00 

PT 

Lead 

Material 

MAT 

PAX 

Material 

0.00 

PT 

Lead 

T  ravel 

TVL 

PAX 

T  ravel 

0.00 

ITT  MILITARY 

PT 

ITT 

James  Reining  (CDR) 

358915,  358911, 
358909 

ML 

PAX 

5.1.3 

GFTD 

1.00 

PT 

ITT 

John  Verniest  (CDR) 

ML 

PAX 

VX-1 

OTD 

1.00 

PT 

ITT 

Kevin  Doney  (CDR) 

359535 

ML 

PAX 

5. 1.6. 3 

1.00 

PT 

ITT 

Saglimbene  (LT) 

359535 

ML 

PAX 

VX-1 

1.00 

PT 

ITT 

Olson  )LT) 

359535 

ML 

PAX 

VX-1 

1.00 

PT 

ITT 

Calloway  (LT) 

359534 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Lewis  (LT) 

359534 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Drake  (LT) 

359534 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Class  136  Pilot 

359533 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Class  136  Pilot 

359533 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Johnson  TJ  (CDR) 

359601 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Lazenka  (LT) 

359601 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Webb  (LT) 

359600 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

Walsh  (LT) 

359600 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

NFO  VX-1 

359599 

ML 

PAX 

VX-1 

1.00 

PT 

ITT 

AW  1 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  2 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  3 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  4 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  5 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  6 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  1 

359540 

ML 

PAX 

5.1.3 

1.00 

PT 

ITT 

AW  2 

359540 

ML 

PAX 

5.1.3 

1.00 

AIRVE 

HICLE 

PT 

AV 

Jason  Allred  (AV) 

98810,  98809, 
98808 

GL 

14 

PAX 

5. 1.6. 3 

Air  Vehicle  T&E  Lead 

0.80 

PT 

AV 

William  Lang  (AV) 

98807,  98806, 
98805 

GL 

13 

PAX 

5.1. 6.3 

Aeromechanics  (Lead) 

0.90 

PT 

AV 

TBD  (AV) 

98807,  98806, 
98806 

GL 

14 

PAX 

5.1. 6.3 

Aeromechanics  Branch  Head  -  USN 

0.00 

PT 

AV 

Jason  Brys  (AV) 

98807,  98806, 
98807 

GL 

13 

PAX 

5.1. 6.3 

Aeromechanics  (Loads) 

0.70 

Figure  20.  Government  ITT  Labor  Profile 
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2. 


Simulation  Approach 


The  approach  was  to  run  the  model  to  obtain  predictions  of  performance  while 
varying  the  resource  levels  and  determining  the  amount  of  time  required  to  complete  all 
the  TWDs.  The  author  modeled  three  alternatives  (A,  B,  and  C)  and  a  baseline  and  the 
results  were  compared.  The  baseline  case  was  the  current  work  force  level.  The  current 
workforce  maintains  existing  manpower  resource  levels  to  draft  and  process  TWDs  while 
they  also  conducted  flight-testing.  This  means  the  members  spend  50  percent  of  their 
time  working  on  TWDs  and  50  percent  of  their  time  conducting  flight- testing.  This 
represented  the  lower  bound  of  resources  and  led  to  the  upper  bound  of  time.  Alternative 
A  maintained  current  staffing  levels  to  process  the  TWDs  and  augmented  the  ITT  to 
conduct  flight-testing.  Alternative  B  augmented  the  ITT  staff  in  known  critical  skill  sets, 
such  as  test  conductors  and  project  test  engineers,  beyond  the  100  percent  staffing  used  in 
Alternative  A  to  work  on  TWDs  and  conduct  testing.  Several  iterations  were  used  before 
a  final  staffing  configuration  was  found  for  Alternative  B  as  described  in  the  next 
paragraph.  Alternative  C  was  constructed  with  unlimited  resources  in  order  to  determine 
the  earliest  time  TWDs  could  be  completed  and  to  identify  the  workforce  needed  to 
accomplish  the  effort  at  that  rate.  This  represented  the  upper  bound  of  resource  and  led 
to  the  lowest  bound  of  time.  The  author  identified  Alternative  C  as  the  “ideal  solution,” 
but  it  carried  a  larger  cost  than  the  other  concepts. 

Early  simulation  runs  of  the  model  supported  sensitivity  analyses  to  detennine  if 
there  were  any  critical  resources.  The  author  performed  the  analyses  by  plotting  task 
completion  time  against  the  number  of  workers.  Examples  of  the  sensitivity  assessment 
for  Boeing  test  conductors,  USN  developmental  test  pilots,  and  USN  mission  systems 
project  engineers  are  shown  in  Figures  21  through  23.  The  assessment  identified  three 
key  resource  elements  that  influenced  TWD  completion.  The  knee  in  the  curve  on  each 
graph  was  used  to  establish  the  optimal  resource  levels.  The  knees  occur  at  the  point 
where  adding  resources  ceases  to  reduce  the  processing  time  significantly.  For  example, 
the  knee  in  Figure  21  for  aircraft  systems  test  conductors  occurs  at  about  the  point  where 
there  are  28  test  conductors.  Adding  more  test  conductors  (beyond  28)  does  not  further 
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reduce  the  processing  time.  Similarly,  the  knees  in  the  curve  for  the  DT  Pilots  and  PE 
Mission  Systems  were  at  11  and  12  workers  respectively.  The  final  Alternative  B 
configuration  was  based  on  this  staffing  level  information. 


Figure  21.  Test  Conductors  Sensitivity  Analysis 


DTPILOT 


Figure  22.  Developmental  Test  Pilots  Sensitivity  Analysis 
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Figure  23.  Mission  Systems  Project  Engineer  Sensitivity  Analysis 

3.  Simulation  Validation 

Figure  24  presents  the  early  simulation  results  used  to  assess  the  model’s 
accuracy.  The  graph  shows  the  model’s  prediction  compared  against  actual  status  for  the 
first  17  TWDs.  These  are  required  to  conduct  testing  on  the  first  P-8  test  aircraft  (T-l). 
The  data  shows  the  number  of  TWDs  completing  the  four  phases  of  the  TWD  process.  It 
compares  the  number  of  TWDs  the  model  projected  with  the  actual  numbers  completed 
in  the  phase.  The  close  agreement  of  the  data  validates  the  model.  The  results  provided 
enough  confidence  that  the  author  updated  the  model  to  run  all  72  TWDs.  These  TWDs 
covered  ground  and  flight-testing  for  the  three  test  aircraft,  T-l,  T-2,  and  T-3.  Table  6 
shows  the  list  of  the  72  TWDs  for  imported  into  the  model. 
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■  Model  Prediction  □  Actuals 


Figure  24.  Early  Simulation  Results 
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Table  6 


TWO  NAME 

Old? 

Compleii" 

Test 

AIRCRAFTVEIGHTBALANCE 

1 

low 

A 

ELECTRICALSYSTEMGROUNDTEST 

2 

low 

B 

FULLSCALEAIRFRAMELOADSCALIBRATION 

3 

medium 

A 

EMCSOFT 

4 

medium 

A 

HUMANF ACTORS 

5 

medium 

A 

GROUNDVIBRATIONTESTCLEANWING 

6 

medium 

A 

STRUCTURALMOOEINTERACTIONCLEANTVD 

7 

medium 

A 

AIRCREVGROUNDEGRESSDEMO 

8 

low 

B 

FQFLIGHTTESTCLEAN 

8 

high 

C 

STRUCTURAILOAOSGROUNDFLIGHTTESTCLEAN 

10 

medium 

C 

RUTTERFLIGHTTESTaEAN 

11 

medium 

c 

TAXIPRELIMINARYFUGHTEVAL 

12 

medium 

c 

PERFORMANCEFLIGHTTESTCLEAN 

13 

medium 

c 

TRAILINGCONESECFLIGHTTEST 

H 

medium 

c 

FUaSYSTEMFUGHTTEST 

15 

medium 

B 

aECTRICALSYSTEMFUGHTTEST 

16 

medium 

B 

AERODYNAMICCONFORMITYGROUNDTEST 

17 

low 

A 

ECSGROUNOTEST 

18 

low 

A 

HEALTHMANAGEMENTSYSTEMFLIGHTTEST 

18 

medium 

A 

CFEW  ALERTING 

20 

low 

A 

ENHANCEDOIGIT  ALFLIGHTCONTROLSEDFCS 

21 

medium 

C 

FLIGHTMANAGEMENTCOMPUTINGSYSTEMFMCS 

22 

medium 

A 

SERYICELIFETRACKtfJGTESTAMIOEMONSTRATIC 

23 

medium 

A 

HEALTHMANAGEMENTSYSTEMGROUNDTESTAN 

24 

medium 

A 

DYNAMICSTORESRELEASE 

25 

medium 

C 

GROUNDHANDLINGTEST 

26 

medium 

B 

FQFLIGHTTESTSTORES 

27 

high 

C 

RAOARFLIGHT 

28 

medium 

A 

EVSPaiGHT 

28 

medium 

B 

PROPULSIONPOVERTHERMALGROUND 

30 

medium 

B 

NOISEVIBEFLIGHTTESTCLEAN 

31 

medium 

C 

AIRCREVGROUNOEGRESSOEMO 

32 

low 

B 

AERIAL  REFUaiNGSIMULATIONGROUNDTEST 

33 

medium 

B 

BUFFET 

34 

high 

C 

OBIGGSGROUNOTEST 

35 

low 

B 

OBIGSSFLIGHTTEST 

36 

medium 

B 

List  of  the  72  TWDS 


TYP 

TVD  NAME 

Oide 

Compleiit 

Test 

TYP 

AS 

FLUTTERFLIGHTTESTSTORES 

37 

medium 

C 

AS 

AS 

PROPULSIONSYSTEMTHERMALPERFORMANCEF 

38 

medium 

C 

AS 

AS 

AERIALREFUELINGFL1GHTTESTTVDF03F002 

39 

high 

C 

AS 

MS 

IFFFLIGHT 

40 

medium 

A 

MS 

MS 

ESMGROUNOFLIGHT 

41 

high 

A 

MS 

AS 

NAVGROUNOFLIGHT 

42 

medium 

A 

MS 

AS 

ECSFLIGHTTEST 

43 

medium 

A 

AS 

AS 

STORESFITCOMPATIB1ITYGROUNDTEST 

44 

medium 

A 

AS 

AS 

STRUCTURALLOAOSFLIGHTTESTSTORES 

45 

medium 

C 

AS 

AS 

INTEGRATEDMISSONSYSTEMOPERATIONSGROU 

46 

high 

A 

MS 

AS 

AIRBORNEEMERGENCYEGRESSDEMONSTRATO 

47 

medium 

C 

AS 

AS 

NOISEVIBEFLIGHTTESTSTORES 

48 

medium 

C 

AS 

AS 

PR0TECTI0NFR0MUNAUTH0RI2EDENTRY0EM0I 

49 

low 

A 

AS 

AS 

CREVINTERIORNOISEANOAUOIBILITYFUGHTTEST 

50 

low 

A 

AS 

AS 

MAINT  AINABILITYOEMONSTRATION 

51 

low 

A 

AS 

AS 

GROUNOANDFLIGHTTRAININGSIMULATORTESTIN 

52 

medium 

A 

AS 

AS 

EOIRFLIGHT 

53 

medium 

A 

MS 

AS 

COMMVOICEGROUNOFLIGHT 

54 

medium 

A 

MS 

AS 

EMCSOFT 

55 

low 

A 

MS 

MS 

CLIMATICCHAMBERGROUNDTEST 

56 

medium 

A 

AS 

AS 

TEMPEST 

57 

low 

A 

MS 

AS 

STORESSTRUCTURALMODEINTERACTIONTEST 

58 

medium 

A 

AS 

AS 

FREEPLAYANDRIGIDITYTEST 

59 

medium 

A 

AS 

AS 

ST  ORESGROUNDVIBR  ATIONTEST 

60 

medium 

A 

AS 

AS 

ACOUSTIC  SGROUNDFLIGHT 

61 

high 

A 

MS 

AS 

WEAPONSINTEGRATIONSMSGRD 

62 

high 

B 

MS 

AS 

VEAPONSFLTANDOELIVERY  ACCURACY 

63 

medium 

C 

MS 

MS 

COMMDAT  ALINKSGROUNOFLIGHT 

64 

medium 

A 

MS 

MS 

EEE 

65 

medium 

A 

MS 

AS 

HUMANF  ACTORSFUU.MISSION 

66 

medium 

B 

MS 

AS 

VEAPONSSEPARATION 

67 

high 

C 

AS 

AS 

ICINGSYSTEMTEST 

68 

medium 

C 

AS 

AS 

VEAPONSDELIVERY  ACCURACY 

69 

medium 

c 

MS 

AS 

AIRCRAFTRESPONSEANDTURNAROUNOTIMEDEr 

70 

low 

A 

AS 

AS 

TAXIPREUMINARYFLIGHTEVALDEMONSTRATION 

71 

medium 

C 

AS 

AS 

PERFORMANCEFLIGHTTESTSTORES 

72 

medium 

c 

AS 
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4.  Improved  TWD  Model 


Since  May  2009,  monthly  simulation  runs  have  been  generated  to  track  the 
accuracy  of  the  model.  Figure  25  is  a  plot  of  the  T-l  TWD  status  for  a  simulation  run 
performed  in  late  July  2009.  The  data  reveals  that  the  model  continues  to  track  well  for 
each  phase  of  the  TWD  process,  confirming  the  value  of  the  tool.  A  small  number  of 
TWDS  were  initiated  back  in  October  2008.  These  TWDs  continued  through  the  process 
and  the  first  one  received  approval  in  January  2009.  The  model’s  start  date  was  January 
2009.  This  difference  of  initiation  date  accounts  for  the  early  differences  on  approved 
TWDs. 


T-1  TWD  Line  of  Balance 


First  Flight 
9/23/09 


Model  Projected 
Actual  Completed 


Figure  25.  Model  Predictions  vs.  Actual  T-l  TWD  Data 
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D.  CHAPTER  SUMMARY 

The  chapter  explained  the  TWD  process  improvements  derived  from  a  functional 
analysis  prior  to  computer  modeling.  It  provided  an  explanation  of  the  four-phase 
process  concept  and  expected  benefits  of  moving  to  new  process. 

This  chapter  outlined  the  search  for  commercially  available  process  modeling 
software  meeting  specific  requirements  associated  with  cost,  software  availability,  and 
ease  of  use.  Results  from  the  research  identified  a  preferred  software  suite  produced  by 
Corel  know  as  iGrafx. 

This  chapter  introduced  the  objective  of  the  TWD  model  and  reviewed  other 
research  applications  in  the  projects  of  Lean  Enterprise  and  NAVAIR  AIRSpeed.  There 
was  a  strong  correlation  with  NAVAIR  and  USAF  depot  maintenance  capacity  models. 
Their  focus  was  on  throughput  of  hardware,  while  the  TWD  model  product  was  the 
creation  of  documentation.  Earlier  projects  were  also  interested  in  understanding  the 
resource  requirements  to  achieve  their  objectives.  These  were  also  products  of  the  TWD 
model  of  interest  to  P-8A  Program  Management. 

This  chapter  described  the  building  of  the  TWD  model.  It  explained  the  various 
entries  as  the  TWD  model  was  assembled.  It  described  the  creation  of  the  software 
model  implementation  of  the  four-phase  approach.  It  identified  data  files  imported  into 
the  model.  Finally,  define  the  factors  modeled  and  the  basis  of  those  factors. 

The  chapter  discussed  how  sensitivity  models  for  labor  categories  were  developed 
and  interpreted  to  determine  critical  resource  categories  and  the  resource  requirements 
needed  to  execute  efficiently  the  TWD  process. 

Finally,  the  chapter  presented  results  from  simulation  runs  conducted  on  the 
current  workforce  and  three  alternatives.  It  compared  the  alternatives  back  to  the 
baseline  to  detennine  the  impact  of  the  various  resource  levels  on  the  TWD  process. 
Early  results  were  validated  with  actual  TWD  production  rate  by  the  ITT.  Results  from 
the  simulations  showed  how  to  allocate  the  resources  to  balance  cost  and  schedule  while 
meeting  the  test  program’s  test  schedule. 
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V.  ANALYSIS  OF  RESULTS 


A.  INTRODUCTION 

The  author  conducted  several  simulation  runs  to  identify  the  necessary  resources 
required  to  move  the  TWDs  through  the  process.  Figure  26  is  the  plot  of  those 
simulation  runs.  The  plot  represents  the  projections  of  the  number  of  TWDs  completed 
for  a  given  resource  level  and  the  date  when  all  TWDs  are  completed.  The  graph  shows 
the  baseline  plan  established  in  April  2008,  and  the  re-baseline  conducted  in  December 
2008.  It  also  displays  the  model’s  prediction  based  on  a  new  start  date,  January  2009  ITT 
staffing  levels,  and  the  three  alternatives.  The  plot  has  three  vertical  lines  labeled  T-l,  T- 
2,  and  T-3.  T-l,  T-2,  T-3  are  designations  for  aircraft  provided  to  support  the  flight  test 
program.  The  T-l,  T-2,  and  T-3  on  the  plot  represents  the  dates  when  the  T-l,  T-2,  and 
T-3  aircraft  commence  flight-test  and  establish  the  target  date  for  TWDs  to  be  completed 
to  support  the  testing.  The  chart  also  has  the  estimated  TWD  execution  rate. 
Approximately  4.3  TWDs  completed  or  executed  is  the  average  monthly  rate.  Six  in  a 
given  month  is  the  estimated  maximum  numbers  of  TWDs  executed.  The  minimum 
number  of  TWDs  in  a  given  month  is  one.  Some  TWDs  like  flutter  or  flying  qualities 
will  be  in  execution  for  several  months  before  being  completed. 

The  plots  of  the  baseline  and  rebaseline  depict  a  very  aggressive  schedule  and 
predicted  all  the  TWDs  would  be  completed  to  support  the  flight  test  program.  However, 
neither  plan  was  staffed  for  this  type  of  execution.  The  model  predicts  that  the  January 
2009  resource  level  and  the  three  alternatives  support  the  date  for  T-l  to  begin  flight¬ 
testing.  However,  the  plots  begin  to  diverge  from  each  other  shortly  after  meeting  the 
initial  need  dates.  Alternative  B  was  identified  as  the  “optimal”  approach  that  completed 
TWDs  by  their  required  test  date  while  balancing  resources.  This  so-called  optimal 
approach  was  detennined  as  the  alternative  for  which  the  cumulative  number  of  TWDs 
delivered  equaled  or  exceeded  the  cumulative  number  of  TWDs  that  could  be  completed 
by  test  aircraft  in  the  actual  flight  test  program. 
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The  most  optimistic  flight  test  program  would  immediately  start  consuming 
TWDs  on  the  test  aircraft  delivery  dates  shown  on  Figure  26  at  the  lines  labeled  T-l,  T-2, 
and  T-3.  Alternative  B  will  meet  the  required  TWD  completion  dates  provided  test 
aircraft  do  not  exceed  an  average  test  completion  rate  of  about  4.3  TWDs  per  month  per 
aircraft. 


TWD  Completion  Dates 


Time 


Figure  26.  Simulation  Results  for  Processing  TWDs  at  Various  Resource  Levels 


Tables  7  and  8  provide  a  summary  of  the  results  for  required  resources.  Table  7 
presents  a  comparison  of  the  resource  levels  of  the  three  concepts  of  alternative  and  the 
baseline.  There  is  a  significant  difference  between  the  baseline  and  Alternative  C.  Table 
8  compares  the  Alternative  B  optimal  workforce  with  the  baseline.  It  also  provides  the 
percent  of  increase  to  resource  to  that  level.  The  study  shows  the  largest  increase  need  is 
for  test  conductors.  This  finding  is  consistent  with  the  situation  in  other  test  programs 
within  NAVAIR.  The  primary  reason  for  this  large  increase  is  that  a  test  conductor  is 
usually  dedicated  to  a  single  TWD  and  oversees  it  from  creation  to  final  approval. 
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Table  7.  Resource  Results  of  the  Alternative  Concepts 


Test  Personnel 

Baseline 

(31  Jan  09 
Resources) 

Alt  A 

AltB 

Alt  C 

TC  AIR 

6 

12 

22 

28 

DT  Pilot 

4 

8 

8 

11 

PE  Air 

11 

22 

22 

34 

OT  Pilot 

5 

10 

15 

15 

TC  MS 

4 

8 

14 

16 

DTNFO 

3 

5 

5 

8 

PE  MS 

5 

9 

11 

12 

OTNFO 

5 

10 

9 

10 

Table  8.  Resource  Requirement  for  the  Improved  TWD  Process 


Test  Personnel 

Baseline 
(31  Jan  09 
Staffing) 

AltB 

(Optimal 

Staffing) 

%Change 

TC  AIR 

6 

22 

266.67% 

DT  Pilot 

4 

8 

100.00% 

PE  Air 

11 

22 

100.00% 

OT  Pilot 

5 

15 

200.00% 

TC  MS 

4 

14 

250.00% 

DTNFO 

3 

5 

66.67% 

PE  MS 

5 

11 

120.00% 

OTNFO 

5 

9 

80.00% 

B.  CHAPTER  SUMMARY 

This  chapter  discussed  the  results  of  the  TWD  model.  It  presented  the  model’s 
baseline  predictions  and  the  three  alternatives.  It  examined  alternatives  that  ranged  from 
a  baseline  condition  that  was  limited  by  the  existing  January  2009  ITT  resource  level  (the 
lower  bound)  to  an  upper  bound  that  assumed  unconstrained  resources.  The  upper  bound 
conditions  were  chosen  to  determine  earliest  time  the  TWDs  could  be  completed.  Other 
options  whose  conditions  were  in  between  those  boundary  conditions  were  also  analyzed 
until  it  was  possible  to  identify  an  “optimal”  resource  level  to  support  the  test  program. 
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An  alternative  (Alternative  B)  was  identified  that  delivered  TWDs  at  rate  such  that  the 
flight-test  schedule  was  not  delayed.  The  selected  alternative  required  the  ITT  resource 
levels  to  increase  by  approximately  150  percent  over  the  baseline.  This  rate  set  staffing 
levels  at  the  appropriate  levels  to  meet  the  schedule,  thus  Alternative  B  is  the  optimal 
solution  among  the  alternatives  considered. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  ability  of  the  test  working  description  (TWD)  model  to  predict  effects  of 
process  changes  has  confirmed  there  is  an  opportunity  to  work  an  “optimal”  solution  of 
balancing  resources  (cost,  personnel),  schedule  and  technical  requirements  (TWDs  in  this 
case).  The  P-8A  test  program  was  having  trouble  writing  test  working  descriptions  (test 
plans)  and  getting  them  approved  soon  enough  to  meet  the  flight  test  schedule.  This 
research  identified  process  improvements  through  computer  modeling  to  predict  TWD 
completion  dates  for  various  resource  levels  and  minor  process  changes  to  improve  TWD 
quality.  The  study  results  in  the  following  conclusions: 

•  The  P-8A  TWD  process  can  be  improved  to  increase  its  efficiency  even 
without  adding  resources.  There  was  an  improvement  in  TWD  quality.  This 
quality  improvement  will  reduce  the  time  a  TWD  remains  in  the  first  phase 
(TPWG). 

•  Modeling  the  TWD  process  to  provide  accurate  and  reliable  results  have  been 
demonstrated.  The  model’s  monthly  forecasts  continue  to  show  strong 
correlation  to  actual  completion  data.  The  ITT  continues  to  use  the  model, 
and  results  have  been  presented  at  program  reviews. 

•  The  model  helped  identify  an  optimal  resource  level  that,  if  implemented, 
would  support  the  P-8A  test  schedule.  Results  from  the  model  showed  the 
resource  level  needed  to  meet  the  test  dates.  The  P-8  A  program  has  reviewed 
results  from  the  model  and  has  been  allocating  resources  to  the  ITT  according 
to  Alternative  B. 

•  There  is  a  strong  interdependence  between  work  force  levels  and  process 
execution  rates.  The  model  results  show  the  relationship  between  TWD 
production  rates  and  the  size  of  the  work  force.  The  model  also  showed  there 
is  a  maximum  throughput  and  increasing  the  work  force  beyond  that  needed  to 
deliver  this  maximum  throughput  has  diminishing  returns. 
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B.  RECOMMENDATIONS 

Results  from  the  TWD  model  of  the  “optimized”  TWD  process  have  been 
validated  based  on  comparison  of  actual  TWD  data  with  the  baseline  alternative 
analyzed.  This  study  recommends  that  the  improved  TWD  process  be  implemented 
within  NAVAIR.  In  addition,  NAVAIR  should  use  the  TWD  computer  process  model  as 
a  tool  to  assess  resource  requirements.  Based  on  application  of  the  model,  Alternative  B 
is  recommended  as  the  optimal  resourcing  to  complete  the  72  TWDs  needed  for  P-8A  test 
program.  These  results  recommend  Alternative  B  for  continued  implementation  within 
the  P-8A  program  in  order  to  achieve  delivery  of  TWDs  in  time  to  support  the  flight  test 
schedule. 

The  optimized  TWD  process  reduced  a  notional  five-phase  process  and  leaned  it 
out  to  a  four-phase  approach  as  shown  in  Figure  17  on  page  66.  The  four-phase  approach 
consists  of:  1)  requirements  review,  2)  first  draft,  3)  technical  readiness  assessment 
(TRA),  and  4)  executive  review  board  (ERB).  The  notional  five-phase  process  had  an 
additional  draft  and  review  between  phases  2  and  3  of  the  improved  four-phase  process. 
Early  coordination  in  phase  1 ,  requirements  review,  allows  for  this  phase  reduction.  The 
team  discovered  that  clear  definitions  of  test  expectations,  roles  and  responsibilities, 
required  dates,  and  interdependencies  to  other  testing  improved  the  quality  of  the  initial 
draft  of  the  TWD. 

C.  APPLICATION  TO  OTHER  NAVAIR  TEST  PROGRAMS 

The  TWD  model  has  flexibility  to  be  adapted  to  any  test  program  within 
NAVAIR.  The  E-2D  program  is  still  in  its  test  phase  and  its  participants  are  actively 
writing  TWDS.  The  E-2D  program  could  benefit  from  the  model’s  ability  to  conduct 
‘what-ifi  drills  to  understand  team  resourcing.  Training  in  use  of  the  model  has  begun 
along  with  assisting  in  shaping  inputs  required  for  the  model. 
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The  NAVAIR  Atlantic  Test  Ranges  program  has  also  expressed  interest  in 
adapting  this  model  to  help  schedule  range  time  and  identify  the  resources  required  to 
support  the  scheduling.  The  Test  Range  has  taken  possession  of  a  copy  of  the  model  for 
further  assessment. 

D.  APPLICATION  TO  OTHER  P-8A  SYSTEMS  ENGINEERING 

PROCESSES 

Two  other  systems  engineering  processes  used  within  the  P-8A  program  are  the 
risk  management  and  the  engineering  change  process.  The  NAVAIR  AIRSpeed  program 
is  analyzing  both  of  these  processes  but  from  a  global  NAVAIR  perspective.  Lessons 
learned  on  improving  the  TWD  process  are  being  shared  with  the  program.  Applying  the 
TWD  model  to  these  two  processes  may  be  possible.  Both  programs  share  the  desire  to 
have  the  optimal  resources  applied  to  provide  a  quality  product  at  the  appointed  time. 
Similar  to  the  E-2D  and  Atlantic  Test  Ranges  adapting  the  model  for  their  use,  the  risk 
and  engineering  change  processes  will  require  an  understanding  of  inputs  and 
assumptions  for  the  model.  Specific  sub-processes  will  require  modifications  but  the 
core  model  will  remain.  Adapting  the  model  for  their  use  awaits  the  results  of  the 
AIRSpeed  project.  That  effort  is  recommended  for  future  research. 
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